Oct 31 19:15:37.302635 ip-10-0-128-104 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 19:15:37.719375 ip-10-0-128-104 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:15:37.719375 ip-10-0-128-104 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 19:15:37.719375 ip-10-0-128-104 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:15:37.719375 ip-10-0-128-104 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:15:37.725114 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725030    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:15:37.725114 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725115    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:15:37.725114 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725121    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725124    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725128    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725131    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725134    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725137    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725140    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725143    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725145    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725148    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725151    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725154    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725156    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725165    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725169    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725172    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725175    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725177    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725180    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725183    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:15:37.725241 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725185    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725188    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725191    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725193    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725196    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725199    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725202    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725204    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725207    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725210    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725213    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725216    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725218    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725222    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725225    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725228    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725230    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725233    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725236    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:15:37.725727 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725238    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725241    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725243    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725246    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725249    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725251    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725255    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725257    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725260    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725263    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725266    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725269    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725272    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725274    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725277    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725279    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725282    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725285    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725288    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:15:37.726229 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725290    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725293    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725296    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725298    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725301    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725303    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725306    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725309    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725312    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725315    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725317    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725320    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725324    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725327    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725330    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725332    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725335    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725354    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725357    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:15:37.726697 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725360    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725363    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725365    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725368    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725371    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725374    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725376    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725379    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725381    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725384    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725387    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725391    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725395    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725398    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725401    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725404    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725407    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725410    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725412    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725415    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:15:37.727155 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725418    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725421    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725424    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725427    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725430    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725432    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.725436    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726291    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726299    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726302    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726306    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726309    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726312    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726315    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726317    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726320    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726322    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726325    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726328    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726330    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:15:37.727667 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726333    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726349    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726354    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726358    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726361    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726363    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726366    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726369    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726372    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726375    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726377    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726380    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726384    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726388    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726391    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726393    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726396    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726399    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726402    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726405    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:15:37.728159 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726408    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726411    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726414    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726416    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726419    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726422    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726425    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726427    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726430    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726432    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726435    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726438    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726441    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726444    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726447    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726449    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726452    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726455    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726457    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726460    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:15:37.728709 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726463    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726465    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726468    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726471    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726475    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726478    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726480    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726483    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726488    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726492    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726495    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726499    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726503    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726506    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726509    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726511    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726514    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726517    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:15:37.729187 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726520    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726522    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726525    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726527    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726530    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726533    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726535    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726538    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726540    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726543    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726545    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726548    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726551    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726553    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726556    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726559    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726561    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726564    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726567    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726570    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:15:37.729638 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726573    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726576    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726579    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726582    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726585    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726587    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726590    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726594    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726596    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726599    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726602    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726604    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726607    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726609    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.726612    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727137    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727142    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727145    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727148    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:15:37.730112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727150    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727153    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727156    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727159    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727162    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727165    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727168    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727171    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727173    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727176    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727178    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727181    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727184    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727187    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727190    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727192    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727195    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727197    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727202    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:15:37.730600 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727206    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727210    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727214    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727216    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727219    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727221    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727224    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727226    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727229    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727231    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727234    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727236    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727239    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727241    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727244    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727246    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727250    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727253    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727257    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727259    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:15:37.731053 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727262    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727265    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727268    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727270    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727274    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727277    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727279    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727282    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727284    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727287    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727290    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727293    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727295    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727298    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727301    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727304    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727307    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727309    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727312    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727314    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:15:37.731661 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727317    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727319    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727322    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727325    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727327    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727330    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727332    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727335    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727352    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727355    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727358    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727360    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727366    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727368    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727372    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727374    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727377    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727380    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727383    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:15:37.732134 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727386    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727389    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727391    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727394    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727397    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727400    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727403    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727405    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727409    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727412    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727414    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727417    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727419    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727422    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727424    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727427    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727429    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727432    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727434    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727437    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:15:37.732642 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727439    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727442    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727444    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.727447    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728583    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728599    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728606    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728611    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728616    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728622    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728627    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728631    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728635    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728638    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728642    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728646    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728649    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728653    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728656    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728659    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728663    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728667    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728672    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728675    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 19:15:37.733114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728678    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728682    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728685    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728690    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728693    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728696    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728699    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728703    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728706    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728709    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728713    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728717    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728721    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728727    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728730    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728733    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728736    2574 flags.go:64] FLAG: --enable-server="true"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728740    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728745    2574 flags.go:64] FLAG: --event-burst="100"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728748    2574 flags.go:64] FLAG: --event-qps="50"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728751    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728754    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728757    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728762    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 19:15:37.733702 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728765    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728768    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728771    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728774    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728777    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728781    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728784    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728787    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728791    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728794    2574 flags.go:64] FLAG: --feature-gates=""
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728801    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728805    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728808    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728811    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728814    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728817    2574 flags.go:64] FLAG: --help="false"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728821    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728824    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728827    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728830    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728834    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728839    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728842    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 19:15:37.734315 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728845    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728849    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728852    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728855    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728858    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728861    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728864    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728867    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728870    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728873    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728876    2574 flags.go:64] FLAG: --lock-file=""
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728879    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728882    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728885    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728891    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728894    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728900    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728904    2574 flags.go:64] FLAG: --logging-format="text"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728907    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728910    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728913    2574 flags.go:64] FLAG: --manifest-url=""
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728916    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728921    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728924    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728928    2574 flags.go:64] FLAG: --max-pods="110"
Oct 31 19:15:37.734899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728931    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728934    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728938    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728942    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728945    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728950    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728953    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728961    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728965    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728968    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728971    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728974    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728977    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728980    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728983    2574 flags.go:64] FLAG: --port="10250"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728986    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728989    2574 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-06ad3bcc727258e01"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728992    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728996    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.728999    2574 flags.go:64] FLAG: --register-node="true"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729002    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729006    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729009    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729012    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729016    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729020    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 19:15:37.735538 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729023    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729026    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729029    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729032    2574 flags.go:64] FLAG: --runonce="false"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729035    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729038    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729041    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729047    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729051    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729054    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729058    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729062    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729065    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729069    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729072    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729076    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729079    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729082    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729085    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729091    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729094    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729097    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729102    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729105    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729108    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 19:15:37.736225 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729111    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729114    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729117    2574 flags.go:64] FLAG: --v="2"
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729123    2574 flags.go:64] FLAG: --version="false"
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729127    2574 flags.go:64] FLAG: --vmodule=""
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729131    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729135    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729278    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729282    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729287    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729290    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729293    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729296    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729299    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729304    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729307    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729310    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729312    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729316    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729319    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729321    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:15:37.736858 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729324    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729327    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729329    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729332    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729335    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729372    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729376    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729380    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729383    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729386    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729389    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729391    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729394    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729397    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729399    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729402    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729404    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729407    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729410    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:15:37.737371 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729413    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729416    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729418    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729421    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729424    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729426    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729430    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729432    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729435    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729438    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729440    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729443    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729447    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729451    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729454    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729458    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729461    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729464    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729466    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:15:37.737876 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729469    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729471    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729474    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729476    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729479    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729481    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729484    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729486    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729489    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729492    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729494    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729496    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729499    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729502    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729505    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729507    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729510    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729512    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729515    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729519    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:15:37.738439 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729522    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729524    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729527    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729530    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729533    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729535    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729538    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729540    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729543    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729546    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729548    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729551    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729553    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729556    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729559    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729561    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729564    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729567    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729569    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:15:37.738932 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729572    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729575    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729577    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729580    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729583    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729585    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729588    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729591    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729593    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729596    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729598    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729601    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729606    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729608    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729613    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.729640    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:15:37.739394 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729676    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729681    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729684    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729687    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729690    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729693    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729696    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729699    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729702    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729705    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729707    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729710    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729713    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729715    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729718    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729720    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729723    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729725    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729728    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729730    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:15:37.739784 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729733    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729736    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729738    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729741    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729744    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729747    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729749    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729752    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729755    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729758    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729760    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729765    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729767    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729770    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729773    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729775    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729778    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729781    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729783    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729786    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:15:37.740261 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729789    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729791    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729794    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729797    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729799    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729802    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729804    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729807    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729809    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729812    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729816    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729819    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729822    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729827    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729830    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729832    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729835    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729838    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729841    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:15:37.740963 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729843    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729846    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729849    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729852    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729856    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729859    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729861    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729864    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729867    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729869    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729872    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729874    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729877    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729879    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729882    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729885    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729888    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729890    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729893    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:15:37.741710 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729896    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729898    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729901    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729903    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729906    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729908    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729911    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729914    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729916    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729919    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729922    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729924    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729927    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729930    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729933    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729936    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729938    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729943    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729946    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729950    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:15:37.742235 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729953    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729957    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729960    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729962    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729965    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729968    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729970    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.729973    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.738077    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.738101    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.738904    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.738911    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.738990    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.738997    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739000    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739003    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739006    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739008    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739012    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:15:37.742752 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739015    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739018    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739022    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739028    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739033    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739038    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739041    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739044    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739047    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739049    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739052    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739055    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739058    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739061    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739063    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739066    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739069    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739072    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739074    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:15:37.743410 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739077    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739079    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739082    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739092    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739095    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739098    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739101    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739105    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739109    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739113    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739117    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739120    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739122    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739125    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739127    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739130    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739132    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739135    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739138    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739140    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:15:37.743927 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739143    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739145    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739148    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739151    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739153    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739156    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739158    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739161    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739163    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739166    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739168    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739171    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739174    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739176    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739179    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739184    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739196    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739200    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739202    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:15:37.744514 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739205    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739207    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739210    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739213    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739215    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739218    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739221    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739230    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739233    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739236    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739238    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739241    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739244    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739246    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739249    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739251    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739254    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739257    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739261    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:15:37.744966 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739265    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739269    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739273    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739276    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739279    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739283    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739287    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739290    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739293    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739297    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739300    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739309    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739312    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739315    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739317    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739320    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739323    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739326    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739328    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:15:37.745427 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739331    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739334    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739351    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.739389    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739601    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739607    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739610    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739613    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739616    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739619    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739621    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739624    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739627    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739630    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739632    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739635    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:15:37.745884 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739637    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739640    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739642    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739645    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739648    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739650    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739654    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739656    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739659    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739668    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739671    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739673    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739676    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739678    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739681    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739684    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739686    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739689    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739691    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739694    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:15:37.746309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739696    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739699    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739701    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739704    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739706    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739709    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739711    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739714    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739716    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739719    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739721    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739724    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739726    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739729    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739731    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739734    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739737    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739739    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739742    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:15:37.746810 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739745    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739747    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739750    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739758    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739761    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739764    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739766    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739770    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739774    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739777    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739779    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739782    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739785    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739788    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739790    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739793    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739795    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739798    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739803    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:15:37.747397 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739807    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739811    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739816    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739818    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739821    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739824    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739826    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739829    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739832    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739835    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739838    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739841    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739844    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739846    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739849    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739851    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739854    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739862    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739865    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:15:37.747891 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739868    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739870    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739873    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739876    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739878    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739880    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739883    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739885    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739888    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739892    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739895    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739898    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739902    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739905    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739907    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739910    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:15:37.748380 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739912    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.739942    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739980    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739985    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739988    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739991    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739994    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.739997    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740000    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740004    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740007    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740010    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740013    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740015    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740018    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740026    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:15:37.748761 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740029    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740032    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740035    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740037    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740039    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740042    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740044    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740047    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740049    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740052    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740054    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740057    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740059    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740062    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740065    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740068    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740070    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740072    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740075    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:15:37.749181 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740077    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740081    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740084    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740090    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740093    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740096    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740099    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740101    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740104    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740106    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740109    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740111    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740135    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740145    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740148    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740151    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740154    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740156    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740160    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:15:37.749660 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740162    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740165    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740167    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740170    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740173    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740175    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740178    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740183    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740187    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740191    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740196    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740199    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740201    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740204    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740207    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740209    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740215    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740218    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740220    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740223    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:15:37.750196 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740225    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740228    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740231    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740233    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740236    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740239    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740241    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740251    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740254    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740256    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740259    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740268    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740271    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740274    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740277    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740279    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740282    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740285    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740287    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740290    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:15:37.750717 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740292    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740295    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740298    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740300    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740303    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740306    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740308    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740311    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740316    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740320    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740323    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740326    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740330    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:37.740332    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.743324    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.745843    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.746963    2574 server.go:1014] "Starting client certificate rotation"
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.747068    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 19:15:37.751422 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.747120    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 19:15:37.751859 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.749328    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:15:37.751859 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.750676    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:15:37.771384 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.771351    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 31 19:15:37.776605 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.776587    2574 log.go:25] "Validated CRI v1 image API"
Oct 31 19:15:37.777592 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.777555    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 19:15:37.780634 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.780603    2574 fs.go:135] Filesystem UUIDs: map[0831209d-8d10-45b6-800e-1fa7fb140c46:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 df679879-3c18-4193-ac99-c272e05e4fc3:/dev/nvme0n1p3]
Oct 31 19:15:37.780735 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.780633    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 19:15:37.786800 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.786640    2574 manager.go:217] Machine: {Timestamp:2026-10-31 19:15:37.785283069 +0000 UTC m=+0.380063287 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099705 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21b719ee2d2e25a4c300e5365301db SystemUUID:ec21b719-ee2d-2e25-a4c3-00e5365301db BootID:29e21b7b-98ac-4ea5-a937-deebc7e2c54e Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ee:ab:5e:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ee:ab:5e:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:a3:7a:46:29:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 19:15:37.787482 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.787466    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 19:15:37.787618 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.787601    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 19:15:37.788928 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.788893    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 19:15:37.789084 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.788933    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-104.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 19:15:37.789775 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.789760    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 19:15:37.789810 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.789778    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 19:15:37.789810 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.789807    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:15:37.789863 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.789821    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:15:37.791141 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.791126    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 19:15:37.796040 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.796020    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 19:15:37.796194 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.796184    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 19:15:37.797056 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.797045    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 19:15:37.797092 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.797060    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 19:15:37.797092 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.797073    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 19:15:37.797092 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.797083    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 31 19:15:37.797092 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.797093    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 19:15:37.797690 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.797661    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:15:37.802653 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.802618    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 19:15:37.810184 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.810161    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:15:37.811256 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.811240    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 19:15:37.811297 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.811280    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 19:15:37.811534 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.811522    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 19:15:37.811574 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.811541    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 19:15:37.811574 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.811551    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 19:15:37.811574 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.811560    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 19:15:37.811574 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.811568    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 19:15:37.811673 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.811576    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 19:15:37.812216 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.812202    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 19:15:37.812216 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.812215    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 19:15:37.812304 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.812226    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 19:15:37.812304 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.812234    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 19:15:37.812304 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.812247    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 19:15:37.812304 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.812260    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 19:15:37.812975 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.812957    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 19:15:37.812975 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.812971    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 19:15:37.817810 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.817789    2574 server.go:1263] "Started kubelet"
Oct 31 19:15:37.817953 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.817863    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 19:15:37.818003 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.817913    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 19:15:37.818045 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.818007    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 19:15:37.818652 ip-10-0-128-104 systemd[1]: Started Kubernetes Kubelet.
Oct 31 19:15:37.819750 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.819591    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 19:15:37.821119 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.821105    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 19:15:37.825799 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:37.825776    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 19:15:37.833667 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.833637    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 19:15:37.835025 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.834996    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 19:15:37.835133 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:37.835113    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:37.835813 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.835799    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 19:15:37.836160 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.836141    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 19:15:37.836160 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.836151    2574 factory.go:55] Registering systemd factory
Oct 31 19:15:37.836291 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.836172    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 31 19:15:37.836291 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.835989    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 19:15:37.836449 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.836332    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 19:15:37.836449 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.836374    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 19:15:37.836716 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.836674    2574 factory.go:153] Registering CRI-O factory
Oct 31 19:15:37.836716 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.836691    2574 factory.go:223] Registration of the crio container factory successfully
Oct 31 19:15:37.836824 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.836746    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 19:15:37.836824 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.836769    2574 factory.go:103] Registering Raw factory
Oct 31 19:15:37.836824 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.836784    2574 manager.go:1199] Started watching for new ooms in manager
Oct 31 19:15:37.837867 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.837849    2574 manager.go:319] Starting recovery of all containers
Oct 31 19:15:37.845574 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.845554    2574 manager.go:324] Recovery completed
Oct 31 19:15:37.853756 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.853728    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:15:37.855789 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.855762    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:15:37.855912 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.855797    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:15:37.855912 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.855809    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:15:37.856335 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.856318    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 31 19:15:37.856400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.856351    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:15:37.856400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.856334    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 19:15:37.856400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.856381    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 19:15:37.857191 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.857176    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:15:37.857247 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.857203    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:15:37.857247 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.857213    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:15:37.858628 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.858615    2574 policy_none.go:50] "Start"
Oct 31 19:15:37.858698 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.858631    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 19:15:37.858698 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.858645    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 19:15:37.860632 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.860613    2574 policy_none.go:44] "Start"
Oct 31 19:15:37.911092 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.908484    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 19:15:37.911092 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.909018    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 19:15:37.911092 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.910030    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 19:15:37.911092 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.910048    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 19:15:37.911092 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.910071    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 19:15:37.911092 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:37.910110    2574 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 19:15:37.912182 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.912165    2574 manager.go:341] "Starting Device Plugin manager"
Oct 31 19:15:37.913113 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:37.913092    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 19:15:37.913113 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.913110    2574 server.go:87] "Starting device plugin registration server"
Oct 31 19:15:37.913353 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.913328    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 19:15:37.913395 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.913354    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 19:15:37.913485 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.913468    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 19:15:37.913598 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.913569    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 19:15:37.913598 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.913583    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 19:15:37.914115 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:37.914089    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 19:15:37.914197 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:37.914146    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:37.948781 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:37.948639    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-104.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 19:15:37.948781 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:37.948649    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-104.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 19:15:37.949880 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:37.948657    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-104.us-west-2.compute.internal.18e3b339e279e953  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-104.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-104.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 19:15:37.817737555 +0000 UTC m=+0.412517778,LastTimestamp:2026-10-31 19:15:37.817737555 +0000 UTC m=+0.412517778,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-104.us-west-2.compute.internal,}"
Oct 31 19:15:38.006551 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:38.006398    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-104.us-west-2.compute.internal.18e3b339e4be6e83  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-104.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-104.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-104.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 19:15:37.855782531 +0000 UTC m=+0.450562748,LastTimestamp:2026-10-31 19:15:37.855782531 +0000 UTC m=+0.450562748,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-104.us-west-2.compute.internal,}"
Oct 31 19:15:38.007238 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:38.007209    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-104.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 19:15:38.007316 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:38.007208    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 19:15:38.007316 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:38.007208    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 19:15:38.011055 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.011033    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"]
Oct 31 19:15:38.011129 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.011110    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:15:38.012062 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.012046    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:15:38.012126 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.012075    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:15:38.012126 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.012086    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:15:38.014183 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.014169    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:15:38.014438 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.014425    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:15:38.014577 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.014560    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.014638 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.014599    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:15:38.014998 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.014981    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:15:38.014998 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.015008    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:15:38.015142 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.015019    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:15:38.015142 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.014983    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:15:38.015142 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.015090    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:15:38.015142 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.015104    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:15:38.015142 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.015126    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.015408 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.015391    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:15:38.015494 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.015422    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:15:38.015494 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.015436    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:15:38.017250 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.017230    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.017354 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.017276    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:15:38.017995 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.017978    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:15:38.018066 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.018004    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:15:38.018066 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.018020    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:15:38.032014 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.031984    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 19:15:38.032159 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.032053    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wtt7p"
Oct 31 19:15:38.054560 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:38.054536    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found" node="ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.057083 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:38.057063    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found" node="ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.096041 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.096006    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:15:38.104867 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.104842    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.104939 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:38.104872    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-104.us-west-2.compute.internal\": node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:38.138542 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.138491    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.138717 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.138549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.138717 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.138623    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70cf5f03a5aad2f217175f2d65705fc5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"70cf5f03a5aad2f217175f2d65705fc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.208309 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:38.208270    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:38.239567 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.239526    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70cf5f03a5aad2f217175f2d65705fc5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"70cf5f03a5aad2f217175f2d65705fc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.239656 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.239572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.239656 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.239597    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.239718 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.239650    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.239718 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.239666    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70cf5f03a5aad2f217175f2d65705fc5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"70cf5f03a5aad2f217175f2d65705fc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:38.239718 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:38.239671    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:39.340532 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:39.340456    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:39.391159 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:39.391129    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:39.394314 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:39.394293    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:39.440625 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:39.440586    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:39.541311 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:39.541279    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:39.641919 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:39.641825    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:39.742837 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:39.742802    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:39.778236 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:39.778201    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 19:15:39.778761 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:39.778404    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 19:15:39.778761 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:39.778399    2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa8143aa486024233b53a2014e030d47-5fe5f50e5295cc6b.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-104.us-west-2.compute.internal.18e3b339e4bee5e5\": read tcp 10.0.128.104:46204->18.224.159.237:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-104.us-west-2.compute.internal.18e3b339e4bee5e5  default   4835 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-104.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-104.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-104.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 19:15:37 +0000 UTC,LastTimestamp:2026-10-31 19:15:38.015022716 +0000 UTC m=+0.609802933,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-104.us-west-2.compute.internal,}"
Oct 31 19:15:39.843728 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:39.843690    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:39.943920 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:39.943858    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:39.995795 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:39.995765    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:15:39.996761 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:39.996741    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:15:39.996869 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:39.996771    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:15:39.996869 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:39.996781    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:15:40.044506 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:40.044474    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:40.069594 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.069558    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:10:38 +0000 UTC" deadline="2028-06-26 04:27:41.336530345 +0000 UTC"
Oct 31 19:15:40.069594 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.069589    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14481h12m1.266944395s"
Oct 31 19:15:40.145021 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:40.144990    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:40.245278 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:40.245205    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:40.345836 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:40.345804    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:40.444904 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.443772    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:15:40.444904 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.444399    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:15:40.445907 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:40.445889    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:15:40.492771 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.492733    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:15:40.567119 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.567087    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:40.625597 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.625572    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:15:40.626391 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.626376    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:15:40.633247 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:40.633219    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83e5200bcb2ab7f96a88eeb2451b2a6.slice/crio-ac545f00e57d82c65d2fbd08eb776c5127df5595567899a4dd1c433595badb75 WatchSource:0}: Error finding container ac545f00e57d82c65d2fbd08eb776c5127df5595567899a4dd1c433595badb75: Status 404 returned error can't find the container with id ac545f00e57d82c65d2fbd08eb776c5127df5595567899a4dd1c433595badb75
Oct 31 19:15:40.633713 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:40.633688    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cf5f03a5aad2f217175f2d65705fc5.slice/crio-d07a0e8ed908aae0ee04fa0954af3ec07c9f6d0f461e0ee4376735304279ee35 WatchSource:0}: Error finding container d07a0e8ed908aae0ee04fa0954af3ec07c9f6d0f461e0ee4376735304279ee35: Status 404 returned error can't find the container with id d07a0e8ed908aae0ee04fa0954af3ec07c9f6d0f461e0ee4376735304279ee35
Oct 31 19:15:40.638478 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.638463    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:15:40.686193 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.686157    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:15:40.830064 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.829984    2574 apiserver.go:52] "Watching apiserver"
Oct 31 19:15:40.862943 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.862917    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:15:40.865957 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.865936    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 19:15:40.932567 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.932530    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 19:15:40.932929 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.932906    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-kttjn","openshift-multus/multus-4njcn","openshift-image-registry/node-ca-t5726","openshift-multus/multus-additional-cni-plugins-gbmm4","openshift-network-operator/iptables-alerter-vg9dn","kube-system/global-pull-secret-syncer-vs8zn","kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-5psz6","openshift-dns/node-resolver-8gxf9","openshift-multus/network-metrics-daemon-nptg8","openshift-network-diagnostics/network-check-target-g5rjx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-xrnqv"]
Oct 31 19:15:40.936762 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.936741    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:40.936874 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:40.936830    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:15:40.938894 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.938877    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:40.938948 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:40.938931    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:15:40.941084 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.941064    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:40.944076 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.944058    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:40.946462 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.946441    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:40.948629 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.948610    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:40.948749 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:40.948686    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:15:40.950831 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.950812    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:40.953444 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.953426    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.955880 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.955862    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.958200 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.958183    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:40.960554 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.960533    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.962818 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.962797    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:40.966911 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.966862    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal" event={"ID":"70cf5f03a5aad2f217175f2d65705fc5","Type":"ContainerStarted","Data":"d07a0e8ed908aae0ee04fa0954af3ec07c9f6d0f461e0ee4376735304279ee35"}
Oct 31 19:15:40.967016 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.966933    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal" event={"ID":"f83e5200bcb2ab7f96a88eeb2451b2a6","Type":"ContainerStarted","Data":"ac545f00e57d82c65d2fbd08eb776c5127df5595567899a4dd1c433595badb75"}
Oct 31 19:15:40.968429 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.968416    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 19:15:40.980944 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.980919    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-var-lib-kubelet\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.981063 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.980949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-tmp\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.981063 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.980974    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-etc-selinux\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:40.981063 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981049    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-dbus\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:40.981207 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981068    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdj6v\" (UniqueName: \"kubernetes.io/projected/9fe381a8-6a81-4e21-96c6-715ecd22ff45-kube-api-access-fdj6v\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:40.981207 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:40.981207 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981125    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvjg\" (UniqueName: \"kubernetes.io/projected/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-kube-api-access-njvjg\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:40.981207 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981153    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwv2\" (UniqueName: \"kubernetes.io/projected/4e36ee6a-f51a-4182-8bfb-68da77946954-kube-api-access-wcwv2\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:40.981207 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981175    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km84r\" (UniqueName: \"kubernetes.io/projected/613da58f-accf-4a50-a1b3-b7dccc52fae7-kube-api-access-km84r\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.981207 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-host\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:40.981207 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981228    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-cnibin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-kubelet\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-systemd-units\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981289    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-slash\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-tuned\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981332    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-device-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981361    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981382    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b05731a8-35c3-4afa-97fa-6d3db50533b3-agent-certs\") pod \"konnectivity-agent-kttjn\" (UID: \"b05731a8-35c3-4afa-97fa-6d3db50533b3\") " pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981405    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-socket-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-iptables-alerter-script\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981478    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-system-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981494    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-conf-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.981546 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-systemd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981564    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-bin\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovn-node-metrics-cert\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tlwl\" (UniqueName: \"kubernetes.io/projected/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kube-api-access-4tlwl\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981675    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-config\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5q49\" (UniqueName: \"kubernetes.io/projected/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-kube-api-access-s5q49\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981727    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-cnibin\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981746    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981800    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-kubelet\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-node-log\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-script-lib\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysconfig\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981897    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-run\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fe381a8-6a81-4e21-96c6-715ecd22ff45-hosts-file\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:40.982203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-multus\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.981978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-etc-kubernetes\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b05731a8-35c3-4afa-97fa-6d3db50533b3-konnectivity-ca\") pod \"konnectivity-agent-kttjn\" (UID: \"b05731a8-35c3-4afa-97fa-6d3db50533b3\") " pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982069    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-sys-fs\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982114    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-system-cni-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-host-slash\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982158    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-socket-dir-parent\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982232    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-modprobe-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982270    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-systemd\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982313    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-netns\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982338    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-etc-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-netd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-registration-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.982901 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982431    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-env-overrides\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-kubernetes\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-sys\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982521    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-k8s-cni-cncf-io\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-netns\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-daemon-config\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-conf\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-lib-modules\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982686    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fe381a8-6a81-4e21-96c6-715ecd22ff45-tmp-dir\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-bin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982741    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6x56\" (UniqueName: \"kubernetes.io/projected/1297e6ef-5035-497c-9777-c6b4448bceba-kube-api-access-f6x56\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982764    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-ovn\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-log-socket\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982820    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-host\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7wg7\" (UniqueName: \"kubernetes.io/projected/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-kube-api-access-g7wg7\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-hostroot\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982924    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-serviceca\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:40.983442 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-var-lib-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.983935 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.982983    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp6kc\" (UniqueName: \"kubernetes.io/projected/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-kube-api-access-xp6kc\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:40.983935 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.983018    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:40.983935 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.983044    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:40.983935 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.983070    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-cni-binary-copy\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.983935 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.983095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:40.983935 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.983119    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-kubelet-config\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:40.983935 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.983144    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-multus-certs\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:40.993088 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.993066    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 19:15:40.993198 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.993066    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 19:15:40.993198 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.993123    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 19:15:40.993198 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.993155    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lwj7k\""
Oct 31 19:15:40.995367 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.995347    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 19:15:40.995367 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.995359    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4ftk8\""
Oct 31 19:15:40.995706 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.995692    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 19:15:40.995706 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.995700    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 19:15:40.995801 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.995720    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 19:15:40.995801 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.995758    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 19:15:40.997871 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.997853    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:15:40.997958 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.997915    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tqdrb\""
Oct 31 19:15:40.997958 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.997939    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 19:15:40.998042 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:40.997969    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 19:15:41.002372 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.002355    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 19:15:41.002446 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.002388    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 19:15:41.002446 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.002400    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2kswh\""
Oct 31 19:15:41.004622 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.004607    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 19:15:41.004885 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.004873    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 19:15:41.004956 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.004943    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 19:15:41.005054 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.005036    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 19:15:41.005087 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.005039    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 19:15:41.005120 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.005086    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 19:15:41.005152 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.005127    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zsjk9\""
Oct 31 19:15:41.006967 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.006951    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 19:15:41.007010 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.006979    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z8492\""
Oct 31 19:15:41.009338 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.009323    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 19:15:41.009597 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.009586    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 19:15:41.009597 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.009592    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d7wnz\""
Oct 31 19:15:41.011753 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.011738    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlrbg\""
Oct 31 19:15:41.011800 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.011755    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:15:41.011853 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.011840    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 19:15:41.013915 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.013903    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vgfpk\""
Oct 31 19:15:41.036205 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.036179    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 19:15:41.036935 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.036921    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 19:15:41.036972 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.036953    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xbxrh"
Oct 31 19:15:41.084277 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084186    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-netns\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.084277 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084222    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-etc-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.084277 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-netd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.084277 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-registration-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084283    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-netns\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084289    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084299    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-etc-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084310    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-netd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084327    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-env-overrides\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-kubernetes\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-sys\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-registration-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084412    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-kubernetes\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-k8s-cni-cncf-io\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084433    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-sys\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-netns\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084440    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-k8s-cni-cncf-io\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-daemon-config\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084481    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-netns\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-conf\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.084527 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-lib-modules\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084535    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fe381a8-6a81-4e21-96c6-715ecd22ff45-tmp-dir\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-bin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084567    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6x56\" (UniqueName: \"kubernetes.io/projected/1297e6ef-5035-497c-9777-c6b4448bceba-kube-api-access-f6x56\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084609    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-ovn\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-log-socket\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084687    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-conf\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084721    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-lib-modules\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084771    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-ovn\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084853    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-log-socket\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.084970    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fe381a8-6a81-4e21-96c6-715ecd22ff45-tmp-dir\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085069    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-bin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085124    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-host\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7wg7\" (UniqueName: \"kubernetes.io/projected/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-kube-api-access-g7wg7\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085163    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-hostroot\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-serviceca\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085201    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-var-lib-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085249    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-var-lib-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.085292 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085257    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-host\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085268    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-hostroot\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp6kc\" (UniqueName: \"kubernetes.io/projected/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-kube-api-access-xp6kc\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085247    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-env-overrides\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-cni-binary-copy\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-kubelet-config\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-multus-certs\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085449    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-var-lib-kubelet\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-var-lib-kubelet\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085526    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-multus-certs\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085531    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-kubelet-config\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085559    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-tmp\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.086140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085605    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-etc-selinux\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-dbus\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdj6v\" (UniqueName: \"kubernetes.io/projected/9fe381a8-6a81-4e21-96c6-715ecd22ff45-kube-api-access-fdj6v\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085697    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085699    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-etc-selinux\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085722    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvjg\" (UniqueName: \"kubernetes.io/projected/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-kube-api-access-njvjg\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwv2\" (UniqueName: \"kubernetes.io/projected/4e36ee6a-f51a-4182-8bfb-68da77946954-kube-api-access-wcwv2\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km84r\" (UniqueName: \"kubernetes.io/projected/613da58f-accf-4a50-a1b3-b7dccc52fae7-kube-api-access-km84r\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085798    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-dbus\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-host\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.085812    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085832    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085837    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-host\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.085865    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs podName:1297e6ef-5035-497c-9777-c6b4448bceba nodeName:}" failed. No retries permitted until 2026-10-31 19:15:41.585850712 +0000 UTC m=+3.149321042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs") pod "network-metrics-daemon-nptg8" (UID: "1297e6ef-5035-497c-9777-c6b4448bceba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085866    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-cnibin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-kubelet\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.086939 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-systemd-units\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-slash\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085953    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-cnibin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-tuned\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085985    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-kubelet\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.085993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-device-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086017    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-systemd-units\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086021    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-slash\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086019    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086061    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b05731a8-35c3-4afa-97fa-6d3db50533b3-agent-certs\") pod \"konnectivity-agent-kttjn\" (UID: \"b05731a8-35c3-4afa-97fa-6d3db50533b3\") " pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086073    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-device-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-socket-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-iptables-alerter-script\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-system-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086171    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086186    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-conf-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.087677 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-systemd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086237    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-bin\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086261    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovn-node-metrics-cert\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086273    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-daemon-config\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086286    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tlwl\" (UniqueName: \"kubernetes.io/projected/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kube-api-access-4tlwl\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086311    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-socket-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-config\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5q49\" (UniqueName: \"kubernetes.io/projected/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-kube-api-access-s5q49\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086380    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-system-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-cnibin\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-kubelet\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086480    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086494    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-node-log\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-script-lib\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.088464 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysconfig\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086571    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-run\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fe381a8-6a81-4e21-96c6-715ecd22ff45-hosts-file\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-multus\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086619    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-bin\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-etc-kubernetes\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086691    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-cni-binary-copy\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086420    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-conf-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b05731a8-35c3-4afa-97fa-6d3db50533b3-konnectivity-ca\") pod \"konnectivity-agent-kttjn\" (UID: \"b05731a8-35c3-4afa-97fa-6d3db50533b3\") " pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.086786    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-sys-fs\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-system-cni-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086821    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-cnibin\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.086836    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret podName:0c39943b-0a43-4184-b616-dda8d9f8545e nodeName:}" failed. No retries permitted until 2026-10-31 19:15:41.586820097 +0000 UTC m=+3.150290421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret") pod "global-pull-secret-syncer-vs8zn" (UID: "0c39943b-0a43-4184-b616-dda8d9f8545e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086877    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-sys-fs\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086892    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-run\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.089211 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086897    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-iptables-alerter-script\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086904    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-host-slash\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086936    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-systemd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086781    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-socket-dir-parent\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-modprobe-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086970    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-etc-kubernetes\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.087000    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-node-log\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.087006    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086989    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.087037    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-socket-dir-parent\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.087048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-kubelet\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.087052    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-multus\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.087017    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fe381a8-6a81-4e21-96c6-715ecd22ff45-hosts-file\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.087476    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysconfig\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.087574    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.087631    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-serviceca\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:41.089931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.087698    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-modprobe-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.090687 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.088017    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-script-lib\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.090687 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.088067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-systemd\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.090687 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.088101    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.090687 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.088169    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-systemd\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.090687 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.088226    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-system-cni-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.090687 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.088398    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-config\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.090687 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.088507    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b05731a8-35c3-4afa-97fa-6d3db50533b3-konnectivity-ca\") pod \"konnectivity-agent-kttjn\" (UID: \"b05731a8-35c3-4afa-97fa-6d3db50533b3\") " pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:41.090687 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.086932    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-host-slash\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:41.091285 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.091265    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-tmp\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.092209 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.092187    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-tuned\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.092497 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.092475    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovn-node-metrics-cert\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.092574 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.092501    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b05731a8-35c3-4afa-97fa-6d3db50533b3-agent-certs\") pod \"konnectivity-agent-kttjn\" (UID: \"b05731a8-35c3-4afa-97fa-6d3db50533b3\") " pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:41.094185 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.094167    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 19:15:41.094229 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.094187    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 19:15:41.141847 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.141811    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:15:41.141847 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.141838    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:15:41.141847 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.141852    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5wfph for pod openshift-network-diagnostics/network-check-target-g5rjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:41.142095 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.141926    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph podName:c70d21f3-352d-45fc-853c-a74f690d1898 nodeName:}" failed. No retries permitted until 2026-10-31 19:15:41.641906153 +0000 UTC m=+3.205376485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5wfph" (UniqueName: "kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph") pod "network-check-target-g5rjx" (UID: "c70d21f3-352d-45fc-853c-a74f690d1898") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:41.142991 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.142957    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6x56\" (UniqueName: \"kubernetes.io/projected/1297e6ef-5035-497c-9777-c6b4448bceba-kube-api-access-f6x56\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:41.143311 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.143287    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwv2\" (UniqueName: \"kubernetes.io/projected/4e36ee6a-f51a-4182-8bfb-68da77946954-kube-api-access-wcwv2\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.143409 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.143312    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tlwl\" (UniqueName: \"kubernetes.io/projected/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kube-api-access-4tlwl\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.143772 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.143750    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5q49\" (UniqueName: \"kubernetes.io/projected/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-kube-api-access-s5q49\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.143862 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.143753    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7wg7\" (UniqueName: \"kubernetes.io/projected/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-kube-api-access-g7wg7\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:41.143862 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.143819    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp6kc\" (UniqueName: \"kubernetes.io/projected/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-kube-api-access-xp6kc\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.144319 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.144293    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvjg\" (UniqueName: \"kubernetes.io/projected/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-kube-api-access-njvjg\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:41.144455 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.144439    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdj6v\" (UniqueName: \"kubernetes.io/projected/9fe381a8-6a81-4e21-96c6-715ecd22ff45-kube-api-access-fdj6v\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:41.144879 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.144860    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km84r\" (UniqueName: \"kubernetes.io/projected/613da58f-accf-4a50-a1b3-b7dccc52fae7-kube-api-access-km84r\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.252940 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.252902    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:15:41.264615 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:41.264588    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccd8e59_1ea3_43ab_bf6e_f2aa341e3a40.slice/crio-3809a62c76782d05a5ab1b565d2b39220002200918ebeebac00247553e17b9f8 WatchSource:0}: Error finding container 3809a62c76782d05a5ab1b565d2b39220002200918ebeebac00247553e17b9f8: Status 404 returned error can't find the container with id 3809a62c76782d05a5ab1b565d2b39220002200918ebeebac00247553e17b9f8
Oct 31 19:15:41.269968 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.269950    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:15:41.276796 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.276778    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:15:41.276886 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.276869    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:15:41.277581 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:41.277560    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bf62e9_8bb7_4a02_96ad_de1985ddb7a4.slice/crio-540a2d1bf2708ed1dd233abc7be4a810bb08e76c08a0d59cecb238d10cb3a063 WatchSource:0}: Error finding container 540a2d1bf2708ed1dd233abc7be4a810bb08e76c08a0d59cecb238d10cb3a063: Status 404 returned error can't find the container with id 540a2d1bf2708ed1dd233abc7be4a810bb08e76c08a0d59cecb238d10cb3a063
Oct 31 19:15:41.287950 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:41.287919    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe381a8_6a81_4e21_96c6_715ecd22ff45.slice/crio-4823da1de91f8837656168f3d4b3a9fb932075e173848fef95de8ac51c0a5d99 WatchSource:0}: Error finding container 4823da1de91f8837656168f3d4b3a9fb932075e173848fef95de8ac51c0a5d99: Status 404 returned error can't find the container with id 4823da1de91f8837656168f3d4b3a9fb932075e173848fef95de8ac51c0a5d99
Oct 31 19:15:41.288494 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.288473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4njcn"
Oct 31 19:15:41.289212 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:41.289192    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e36ee6a_f51a_4182_8bfb_68da77946954.slice/crio-7b253487198586d7fe47bf2bad1a818744ac7de28b2a03b8843a2637d06345e4 WatchSource:0}: Error finding container 7b253487198586d7fe47bf2bad1a818744ac7de28b2a03b8843a2637d06345e4: Status 404 returned error can't find the container with id 7b253487198586d7fe47bf2bad1a818744ac7de28b2a03b8843a2637d06345e4
Oct 31 19:15:41.290411 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.290396    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:41.297619 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.297593    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:15:41.298051 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:41.298028    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613da58f_accf_4a50_a1b3_b7dccc52fae7.slice/crio-9d38d9fda59af58fef3810194e579b4f6d68ea4c7405dfc86ba96eb8d913fb00 WatchSource:0}: Error finding container 9d38d9fda59af58fef3810194e579b4f6d68ea4c7405dfc86ba96eb8d913fb00: Status 404 returned error can't find the container with id 9d38d9fda59af58fef3810194e579b4f6d68ea4c7405dfc86ba96eb8d913fb00
Oct 31 19:15:41.299564 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.299545    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:15:41.300924 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:41.300902    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05731a8_35c3_4afa_97fa_6d3db50533b3.slice/crio-a2e6eed54b1b98f5f4f90c251ba8ed62f3164a28611fcd0f351f42159ecf8c38 WatchSource:0}: Error finding container a2e6eed54b1b98f5f4f90c251ba8ed62f3164a28611fcd0f351f42159ecf8c38: Status 404 returned error can't find the container with id a2e6eed54b1b98f5f4f90c251ba8ed62f3164a28611fcd0f351f42159ecf8c38
Oct 31 19:15:41.305772 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.305751    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:15:41.308112 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:41.308092    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c72eeb1_17c0_4dd4_99c3_2509ba36b668.slice/crio-a4040f38b2d991ed01cde381142f55f8cb1886d0d151d3b704c0aad4cf754caf WatchSource:0}: Error finding container a4040f38b2d991ed01cde381142f55f8cb1886d0d151d3b704c0aad4cf754caf: Status 404 returned error can't find the container with id a4040f38b2d991ed01cde381142f55f8cb1886d0d151d3b704c0aad4cf754caf
Oct 31 19:15:41.311001 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:41.310978    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d6c1fe_fc02_4e60_9d54_3b66eb513ed7.slice/crio-ec62fe4b4ee9e4f55de91bf1e284cb6d86fccfcd4ef67261b180f2f925a735be WatchSource:0}: Error finding container ec62fe4b4ee9e4f55de91bf1e284cb6d86fccfcd4ef67261b180f2f925a735be: Status 404 returned error can't find the container with id ec62fe4b4ee9e4f55de91bf1e284cb6d86fccfcd4ef67261b180f2f925a735be
Oct 31 19:15:41.315448 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:15:41.315429    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37ff40c_3db6_4467_aeaa_c23407b5fce5.slice/crio-b5739c8cbd0f5e746ddb218c574abe4d33dc870cbb2084e4effce2ddb298fc53 WatchSource:0}: Error finding container b5739c8cbd0f5e746ddb218c574abe4d33dc870cbb2084e4effce2ddb298fc53: Status 404 returned error can't find the container with id b5739c8cbd0f5e746ddb218c574abe4d33dc870cbb2084e4effce2ddb298fc53
Oct 31 19:15:41.591127 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.591098    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:41.591267 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.591139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:41.591267 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.591245    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:41.591267 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.591251    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:41.591374 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.591295    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret podName:0c39943b-0a43-4184-b616-dda8d9f8545e nodeName:}" failed. No retries permitted until 2026-10-31 19:15:42.5912838 +0000 UTC m=+4.154754123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret") pod "global-pull-secret-syncer-vs8zn" (UID: "0c39943b-0a43-4184-b616-dda8d9f8545e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:41.591374 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.591311    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs podName:1297e6ef-5035-497c-9777-c6b4448bceba nodeName:}" failed. No retries permitted until 2026-10-31 19:15:42.591304199 +0000 UTC m=+4.154774509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs") pod "network-metrics-daemon-nptg8" (UID: "1297e6ef-5035-497c-9777-c6b4448bceba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:41.692115 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.692083    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:41.692289 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.692249    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:15:41.692336 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.692289    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:15:41.692336 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.692305    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5wfph for pod openshift-network-diagnostics/network-check-target-g5rjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:41.692418 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:41.692366    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph podName:c70d21f3-352d-45fc-853c-a74f690d1898 nodeName:}" failed. No retries permitted until 2026-10-31 19:15:42.692349343 +0000 UTC m=+4.255819850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wfph" (UniqueName: "kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph") pod "network-check-target-g5rjx" (UID: "c70d21f3-352d-45fc-853c-a74f690d1898") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:41.952014 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.951925    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv" event={"ID":"1c72eeb1-17c0-4dd4-99c3-2509ba36b668","Type":"ContainerStarted","Data":"a4040f38b2d991ed01cde381142f55f8cb1886d0d151d3b704c0aad4cf754caf"}
Oct 31 19:15:41.954617 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.954589    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kttjn" event={"ID":"b05731a8-35c3-4afa-97fa-6d3db50533b3","Type":"ContainerStarted","Data":"a2e6eed54b1b98f5f4f90c251ba8ed62f3164a28611fcd0f351f42159ecf8c38"}
Oct 31 19:15:41.956141 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.956113    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t5726" event={"ID":"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40","Type":"ContainerStarted","Data":"3809a62c76782d05a5ab1b565d2b39220002200918ebeebac00247553e17b9f8"}
Oct 31 19:15:41.963693 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.962362    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l" event={"ID":"b37ff40c-3db6-4467-aeaa-c23407b5fce5","Type":"ContainerStarted","Data":"b5739c8cbd0f5e746ddb218c574abe4d33dc870cbb2084e4effce2ddb298fc53"}
Oct 31 19:15:41.973238 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.973187    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5psz6" event={"ID":"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7","Type":"ContainerStarted","Data":"ec62fe4b4ee9e4f55de91bf1e284cb6d86fccfcd4ef67261b180f2f925a735be"}
Oct 31 19:15:41.974720 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.974687    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4njcn" event={"ID":"613da58f-accf-4a50-a1b3-b7dccc52fae7","Type":"ContainerStarted","Data":"9d38d9fda59af58fef3810194e579b4f6d68ea4c7405dfc86ba96eb8d913fb00"}
Oct 31 19:15:41.976415 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.976369    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbmm4" event={"ID":"4e36ee6a-f51a-4182-8bfb-68da77946954","Type":"ContainerStarted","Data":"7b253487198586d7fe47bf2bad1a818744ac7de28b2a03b8843a2637d06345e4"}
Oct 31 19:15:41.978278 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.977973    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8gxf9" event={"ID":"9fe381a8-6a81-4e21-96c6-715ecd22ff45","Type":"ContainerStarted","Data":"4823da1de91f8837656168f3d4b3a9fb932075e173848fef95de8ac51c0a5d99"}
Oct 31 19:15:41.979356 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:41.979330    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vg9dn" event={"ID":"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4","Type":"ContainerStarted","Data":"540a2d1bf2708ed1dd233abc7be4a810bb08e76c08a0d59cecb238d10cb3a063"}
Oct 31 19:15:42.039752 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:42.039680    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:10:40 +0000 UTC" deadline="2028-05-24 00:53:30.553830551 +0000 UTC"
Oct 31 19:15:42.039752 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:42.039745    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13685h37m48.514088899s"
Oct 31 19:15:42.600791 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:42.600745    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:42.600988 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:42.600844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:42.600988 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.600963    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:42.601104 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.601021    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs podName:1297e6ef-5035-497c-9777-c6b4448bceba nodeName:}" failed. No retries permitted until 2026-10-31 19:15:44.601001429 +0000 UTC m=+6.164471745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs") pod "network-metrics-daemon-nptg8" (UID: "1297e6ef-5035-497c-9777-c6b4448bceba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:42.601104 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.601084    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:42.601213 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.601115    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret podName:0c39943b-0a43-4184-b616-dda8d9f8545e nodeName:}" failed. No retries permitted until 2026-10-31 19:15:44.601104635 +0000 UTC m=+6.164574948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret") pod "global-pull-secret-syncer-vs8zn" (UID: "0c39943b-0a43-4184-b616-dda8d9f8545e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:42.701486 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:42.701452    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:42.701698 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.701612    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:15:42.701698 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.701634    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:15:42.701698 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.701663    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5wfph for pod openshift-network-diagnostics/network-check-target-g5rjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:42.701867 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.701729    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph podName:c70d21f3-352d-45fc-853c-a74f690d1898 nodeName:}" failed. No retries permitted until 2026-10-31 19:15:44.701709062 +0000 UTC m=+6.265179376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wfph" (UniqueName: "kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph") pod "network-check-target-g5rjx" (UID: "c70d21f3-352d-45fc-853c-a74f690d1898") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:42.944741 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:42.942694    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:42.944741 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.942829    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:15:42.944741 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:42.942941    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:42.944741 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.943091    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:15:42.944741 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:42.943160    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:42.944741 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:42.943230    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:15:43.040262 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:43.040190    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:10:40 +0000 UTC" deadline="2028-06-25 19:05:41.733867453 +0000 UTC"
Oct 31 19:15:43.040262 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:43.040229    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14471h49m58.693642904s"
Oct 31 19:15:44.628316 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:44.628216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:44.628316 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:44.628311    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:44.628779 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.628441    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:44.628779 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.628507    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs podName:1297e6ef-5035-497c-9777-c6b4448bceba nodeName:}" failed. No retries permitted until 2026-10-31 19:15:48.628488043 +0000 UTC m=+10.191958374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs") pod "network-metrics-daemon-nptg8" (UID: "1297e6ef-5035-497c-9777-c6b4448bceba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:44.628779 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.628577    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:44.628779 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.628612    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret podName:0c39943b-0a43-4184-b616-dda8d9f8545e nodeName:}" failed. No retries permitted until 2026-10-31 19:15:48.628600283 +0000 UTC m=+10.192070607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret") pod "global-pull-secret-syncer-vs8zn" (UID: "0c39943b-0a43-4184-b616-dda8d9f8545e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:44.729579 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:44.729545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:44.729808 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.729787    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:15:44.729859 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.729814    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:15:44.729859 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.729828    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5wfph for pod openshift-network-diagnostics/network-check-target-g5rjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:44.729924 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.729896    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph podName:c70d21f3-352d-45fc-853c-a74f690d1898 nodeName:}" failed. No retries permitted until 2026-10-31 19:15:48.729876665 +0000 UTC m=+10.293346997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wfph" (UniqueName: "kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph") pod "network-check-target-g5rjx" (UID: "c70d21f3-352d-45fc-853c-a74f690d1898") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:44.942219 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:44.941752    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:44.942219 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.941896    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:15:44.942219 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:44.941977    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:44.942219 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.942049    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:15:44.942219 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:44.942112    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:44.942219 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:44.942178    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:15:45.017237 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:45.017183    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal" event={"ID":"70cf5f03a5aad2f217175f2d65705fc5","Type":"ContainerStarted","Data":"05345beb120d205be3ea9e0d4a7537b36155978c2dc9f991271f2408ed9e94ab"}
Oct 31 19:15:45.023982 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:45.023302    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbmm4" event={"ID":"4e36ee6a-f51a-4182-8bfb-68da77946954","Type":"ContainerStarted","Data":"18a41580961d6137b1c98cea9e6f03f4fc7996686fc56abcf5a747653755e329"}
Oct 31 19:15:45.026310 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:45.026280    2574 generic.go:356] "Generic (PLEG): container finished" podID="f83e5200bcb2ab7f96a88eeb2451b2a6" containerID="15b044bd6ec1b426d8e419feec768a4dcd0858e673ffe50abd2db2bf71a8478a" exitCode=0
Oct 31 19:15:45.026455 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:45.026324    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal" event={"ID":"f83e5200bcb2ab7f96a88eeb2451b2a6","Type":"ContainerDied","Data":"15b044bd6ec1b426d8e419feec768a4dcd0858e673ffe50abd2db2bf71a8478a"}
Oct 31 19:15:45.130055 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:45.129990    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal" podStartSLOduration=5.129976265 podStartE2EDuration="5.129976265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:45.129631761 +0000 UTC m=+6.693102094" watchObservedRunningTime="2026-10-31 19:15:45.129976265 +0000 UTC m=+6.693446596"
Oct 31 19:15:46.033510 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:46.032661    2574 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="18a41580961d6137b1c98cea9e6f03f4fc7996686fc56abcf5a747653755e329" exitCode=0
Oct 31 19:15:46.033510 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:46.032767    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbmm4" event={"ID":"4e36ee6a-f51a-4182-8bfb-68da77946954","Type":"ContainerDied","Data":"18a41580961d6137b1c98cea9e6f03f4fc7996686fc56abcf5a747653755e329"}
Oct 31 19:15:46.043775 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:46.043693    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal" event={"ID":"f83e5200bcb2ab7f96a88eeb2451b2a6","Type":"ContainerStarted","Data":"1473f7eeee0a5232f33be8509b7ade7b7be90126d5af393ad8d2f0bda37316e5"}
Oct 31 19:15:46.943665 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:46.942092    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:46.943665 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:46.942245    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:15:46.943665 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:46.942323    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:46.943665 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:46.942423    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:15:46.943665 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:46.942484    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:46.943665 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:46.942567    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:15:47.055044 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:47.054976    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kttjn" event={"ID":"b05731a8-35c3-4afa-97fa-6d3db50533b3","Type":"ContainerStarted","Data":"945e047df3da7f45e53e01db442c30e11f879478e7fabf5aced472945fd35fef"}
Oct 31 19:15:47.168425 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:47.168351    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal" podStartSLOduration=7.168331077 podStartE2EDuration="7.168331077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:46.279747802 +0000 UTC m=+7.843218133" watchObservedRunningTime="2026-10-31 19:15:47.168331077 +0000 UTC m=+8.731801502"
Oct 31 19:15:47.168719 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:47.168686    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kttjn" podStartSLOduration=3.3581697569999998 podStartE2EDuration="8.168676897s" totalImagesPullingTime="4.81050714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:47.168026053 +0000 UTC m=+8.731496385" watchObservedRunningTime="2026-10-31 19:15:47.168676897 +0000 UTC m=+8.732147228"
Oct 31 19:15:48.060762 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:48.060630    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l" event={"ID":"b37ff40c-3db6-4467-aeaa-c23407b5fce5","Type":"ContainerStarted","Data":"2a95d8c4cefab9add61e25af1b52e8daee557d5836a2df0046ac1018527b0098"}
Oct 31 19:15:48.064637 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:48.064597    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t5726" event={"ID":"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40","Type":"ContainerStarted","Data":"eaa6e53a5e1def16dad1fd74b65f57dea9d6413ed7b82cb2954feba04425a0a9"}
Oct 31 19:15:48.666738 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:48.666701    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:48.666928 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:48.666812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:48.666996 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.666967    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:48.667055 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.667045    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs podName:1297e6ef-5035-497c-9777-c6b4448bceba nodeName:}" failed. No retries permitted until 2026-10-31 19:15:56.667028488 +0000 UTC m=+18.230498797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs") pod "network-metrics-daemon-nptg8" (UID: "1297e6ef-5035-497c-9777-c6b4448bceba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:48.667117 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.667097    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:48.667168 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.667120    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret podName:0c39943b-0a43-4184-b616-dda8d9f8545e nodeName:}" failed. No retries permitted until 2026-10-31 19:15:56.667113352 +0000 UTC m=+18.230583661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret") pod "global-pull-secret-syncer-vs8zn" (UID: "0c39943b-0a43-4184-b616-dda8d9f8545e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:48.767915 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:48.767848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:48.768152 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.768098    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:15:48.768152 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.768123    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:15:48.768152 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.768143    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5wfph for pod openshift-network-diagnostics/network-check-target-g5rjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:48.768280 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.768204    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph podName:c70d21f3-352d-45fc-853c-a74f690d1898 nodeName:}" failed. No retries permitted until 2026-10-31 19:15:56.768186301 +0000 UTC m=+18.331656612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wfph" (UniqueName: "kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph") pod "network-check-target-g5rjx" (UID: "c70d21f3-352d-45fc-853c-a74f690d1898") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:48.943020 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:48.942953    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:48.943149 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.943036    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:15:48.943149 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:48.943085    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:48.943149 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.943127    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:15:48.943250 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:48.943166    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:48.943250 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:48.943208    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:15:49.904913 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:49.904882    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 19:15:49.954106 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:49.953993    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:15:49.904905676Z","UUID":"1eff79a6-bf5b-4af4-8071-ef32755f20d0","Handler":null,"Name":"","Endpoint":""}
Oct 31 19:15:49.957028 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:49.957006    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 19:15:49.957028 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:49.957033    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 19:15:50.072643 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:50.072535    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l" event={"ID":"b37ff40c-3db6-4467-aeaa-c23407b5fce5","Type":"ContainerStarted","Data":"be9b54746b613d941de8a4efffb9710e19c3dadaf0dfb1aad8f52ff3421f8ff6"}
Oct 31 19:15:50.075009 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:50.074313    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4njcn" event={"ID":"613da58f-accf-4a50-a1b3-b7dccc52fae7","Type":"ContainerStarted","Data":"fe965ef5043fba53c84926d8c9958f548608173d9c1c478caa696f74599b317e"}
Oct 31 19:15:50.188856 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:50.188804    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t5726" podStartSLOduration=5.471434899 podStartE2EDuration="11.188789863s" totalImagesPullingTime="5.717354964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:48.178782206 +0000 UTC m=+9.742252538" watchObservedRunningTime="2026-10-31 19:15:50.188789863 +0000 UTC m=+11.752260194"
Oct 31 19:15:50.946489 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:50.946458    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:50.946969 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:50.946590    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:50.946969 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:50.946599    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:15:50.946969 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:50.946608    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:50.946969 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:50.946731    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:15:50.946969 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:50.946837    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:15:51.079374 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:51.079332    2574 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="80cea43c8f9da61662e0d6b92bedcb2dc6231282a84820b4c9f6fc6c5af28104" exitCode=0
Oct 31 19:15:51.079566 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:51.079425    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbmm4" event={"ID":"4e36ee6a-f51a-4182-8bfb-68da77946954","Type":"ContainerDied","Data":"80cea43c8f9da61662e0d6b92bedcb2dc6231282a84820b4c9f6fc6c5af28104"}
Oct 31 19:15:51.081556 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:51.081496    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8gxf9" event={"ID":"9fe381a8-6a81-4e21-96c6-715ecd22ff45","Type":"ContainerStarted","Data":"c9d84b4e1b00ee37f2e49e7691ba10d5f5fa8d8b685fd2ba2ea0e397f74ed751"}
Oct 31 19:15:51.195931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:51.195865    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4njcn" podStartSLOduration=4.064241859 podStartE2EDuration="12.195846021s" totalImagesPullingTime="8.131604162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:50.190789185 +0000 UTC m=+11.754259517" watchObservedRunningTime="2026-10-31 19:15:51.195846021 +0000 UTC m=+12.759316354"
Oct 31 19:15:51.307800 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:51.307764    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:51.308456 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:51.308435    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:51.308830 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:51.308781    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8gxf9" podStartSLOduration=3.654963554 podStartE2EDuration="12.308764309s" totalImagesPullingTime="8.653800755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:51.308732456 +0000 UTC m=+12.872202918" watchObservedRunningTime="2026-10-31 19:15:51.308764309 +0000 UTC m=+12.872234641"
Oct 31 19:15:52.085245 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:52.085203    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vg9dn" event={"ID":"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4","Type":"ContainerStarted","Data":"2d440b76ec5a99c2bd0ce21d155fca55e2e5046dea863f73ed695e2c922233ae"}
Oct 31 19:15:52.209324 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:52.209268    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vg9dn" podStartSLOduration=4.654300061 podStartE2EDuration="13.209255812s" totalImagesPullingTime="8.554955751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:52.208692169 +0000 UTC m=+13.772162503" watchObservedRunningTime="2026-10-31 19:15:52.209255812 +0000 UTC m=+13.772726166"
Oct 31 19:15:52.942007 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:52.941977    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:52.942007 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:52.942001    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:52.942240 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:52.942125    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:15:52.942412 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:52.942395    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:52.942499 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:52.942480    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:15:52.942550 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:52.942541    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:15:53.089752 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:53.089665    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l" event={"ID":"b37ff40c-3db6-4467-aeaa-c23407b5fce5","Type":"ContainerStarted","Data":"5241eee6c6b550918becdecdfe9384b48b5e916864ac51dc169c978b59c4e8b7"}
Oct 31 19:15:53.091114 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:53.091086    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5psz6" event={"ID":"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7","Type":"ContainerStarted","Data":"bdd2bd446c08aa4bc8960f09f5ff10d72e46e959204ca67fa75abc312cacb06f"}
Oct 31 19:15:53.093321 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:53.093292    2574 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="b18580c3130edaa8c2f6810571c899bb5b63ffb9445d4681e1f94ab1bebb0013" exitCode=0
Oct 31 19:15:53.093440 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:53.093378    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbmm4" event={"ID":"4e36ee6a-f51a-4182-8bfb-68da77946954","Type":"ContainerDied","Data":"b18580c3130edaa8c2f6810571c899bb5b63ffb9445d4681e1f94ab1bebb0013"}
Oct 31 19:15:53.203539 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:53.203480    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l" podStartSLOduration=3.670690165 podStartE2EDuration="14.203463698s" totalImagesPullingTime="10.532773533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:53.203104166 +0000 UTC m=+14.766574533" watchObservedRunningTime="2026-10-31 19:15:53.203463698 +0000 UTC m=+14.766934041"
Oct 31 19:15:53.318699 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:53.318629    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5psz6" podStartSLOduration=3.373457852 podStartE2EDuration="14.3186143s" totalImagesPullingTime="10.945156448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:53.31836869 +0000 UTC m=+14.881839021" watchObservedRunningTime="2026-10-31 19:15:53.3186143 +0000 UTC m=+14.882084628"
Oct 31 19:15:54.705482 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:54.705438    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:54.706122 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:54.705624    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 19:15:54.706329 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:54.706304    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:15:54.942747 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:54.942560    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:54.942874 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:54.942835    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:15:54.942931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:54.942904    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:54.942977 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:54.942961    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:15:54.943136 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:54.943114    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:54.943263 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:54.943210    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:15:55.101618 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:55.101568    2574 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="41c9331b1b906a9a6a359852fd9a49065dac681151065120e052bd2efc154f13" exitCode=0
Oct 31 19:15:55.101787 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:55.101629    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbmm4" event={"ID":"4e36ee6a-f51a-4182-8bfb-68da77946954","Type":"ContainerDied","Data":"41c9331b1b906a9a6a359852fd9a49065dac681151065120e052bd2efc154f13"}
Oct 31 19:15:56.733120 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:56.733085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:56.733676 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:56.733154    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:56.733676 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.733236    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:56.733676 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.733250    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:56.733676 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.733304    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret podName:0c39943b-0a43-4184-b616-dda8d9f8545e nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.733287091 +0000 UTC m=+34.296757414 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret") pod "global-pull-secret-syncer-vs8zn" (UID: "0c39943b-0a43-4184-b616-dda8d9f8545e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:15:56.733676 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.733320    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs podName:1297e6ef-5035-497c-9777-c6b4448bceba nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.733312989 +0000 UTC m=+34.296783334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs") pod "network-metrics-daemon-nptg8" (UID: "1297e6ef-5035-497c-9777-c6b4448bceba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:15:56.834444 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:56.834404    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:56.834615 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.834595    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:15:56.834724 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.834622    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:15:56.834724 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.834635    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5wfph for pod openshift-network-diagnostics/network-check-target-g5rjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:56.834801 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.834725    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph podName:c70d21f3-352d-45fc-853c-a74f690d1898 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.834705597 +0000 UTC m=+34.398175919 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wfph" (UniqueName: "kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph") pod "network-check-target-g5rjx" (UID: "c70d21f3-352d-45fc-853c-a74f690d1898") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:15:56.941696 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:56.941643    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:56.941878 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:56.941643    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:56.941878 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:56.941788    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:56.941878 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.941786    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:15:56.942037 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.941889    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:15:56.942037 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:56.941971    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:15:58.112105 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:58.112067    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv" event={"ID":"1c72eeb1-17c0-4dd4-99c3-2509ba36b668","Type":"ContainerStarted","Data":"6e9d26b6685f126d436da1e7f84a0d9bd06f0fb24518d08dcb9e718fb4f308b0"}
Oct 31 19:15:58.112569 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:58.112205    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv" event={"ID":"1c72eeb1-17c0-4dd4-99c3-2509ba36b668","Type":"ContainerStarted","Data":"6f3a6dbb266ea38786dee59f6b7e979dc16fe85ba5af585357f62b534fc18626"}
Oct 31 19:15:58.112569 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:58.112260    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv" event={"ID":"1c72eeb1-17c0-4dd4-99c3-2509ba36b668","Type":"ContainerStarted","Data":"28f623d72f8dcf88baa3a52f4a6911062016a9f97a23838141cee1e2ef5054a7"}
Oct 31 19:15:58.112569 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:58.112295    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv" event={"ID":"1c72eeb1-17c0-4dd4-99c3-2509ba36b668","Type":"ContainerStarted","Data":"1e6f0269465c1ab06b6c5590557704fd3c9a2a5f857e4bf631bde0759aba2e5c"}
Oct 31 19:15:58.112569 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:58.112328    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv" event={"ID":"1c72eeb1-17c0-4dd4-99c3-2509ba36b668","Type":"ContainerStarted","Data":"d5c8ac67471ca7b9239b0db790d707bee4562e206ed30888f7c0676cacc81f52"}
Oct 31 19:15:58.112569 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:58.112361    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv" event={"ID":"1c72eeb1-17c0-4dd4-99c3-2509ba36b668","Type":"ContainerStarted","Data":"740364d51647e4ca752a819fb77251f1c853bd4931b027e745e3acd28303df83"}
Oct 31 19:15:58.942672 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:58.942612    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:15:58.942866 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:58.942766    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:15:58.942866 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:58.942784    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:15:58.942866 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:58.942854    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:15:58.943034 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:58.942912    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:15:58.943034 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:15:58.942993    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:15:59.117157 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:59.117116    2574 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="40fb9dddcdf07daf1f08629419fcf039326eb81c3d51504c63649da51c0ecfb2" exitCode=0
Oct 31 19:15:59.117886 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:15:59.117176    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbmm4" event={"ID":"4e36ee6a-f51a-4182-8bfb-68da77946954","Type":"ContainerDied","Data":"40fb9dddcdf07daf1f08629419fcf039326eb81c3d51504c63649da51c0ecfb2"}
Oct 31 19:16:00.121920 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:00.121886    2574 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="b37eb1661ffba7ce4abe7e1f1f6607c65df035a2c0d6c1202ac3550fbc358e18" exitCode=0
Oct 31 19:16:00.122342 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:00.121962    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbmm4" event={"ID":"4e36ee6a-f51a-4182-8bfb-68da77946954","Type":"ContainerDied","Data":"b37eb1661ffba7ce4abe7e1f1f6607c65df035a2c0d6c1202ac3550fbc358e18"}
Oct 31 19:16:00.125420 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:00.125395    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv" event={"ID":"1c72eeb1-17c0-4dd4-99c3-2509ba36b668","Type":"ContainerStarted","Data":"05ac49b5faa2cdd722c7172c2810239c9c8670e2bf2079cd9add8dc611955f01"}
Oct 31 19:16:00.942398 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:00.942360    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:00.942398 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:00.942382    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:00.942629 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:00.942530    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:16:00.942629 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:00.942576    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:00.942761 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:00.942706    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:16:00.942806 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:00.942783    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:16:01.131347 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:01.131308    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbmm4" event={"ID":"4e36ee6a-f51a-4182-8bfb-68da77946954","Type":"ContainerStarted","Data":"cf980506c68eef79af9109fb93f1bcfee6d061a19f524ba84760ed3f1e3b7e27"}
Oct 31 19:16:01.296765 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:01.296638    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gbmm4" podStartSLOduration=7.806660009 podStartE2EDuration="22.296622343s" totalImagesPullingTime="14.489962334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:01.296423755 +0000 UTC m=+22.859894086" watchObservedRunningTime="2026-10-31 19:16:01.296622343 +0000 UTC m=+22.860092665"
Oct 31 19:16:02.137317 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:02.137279    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv" event={"ID":"1c72eeb1-17c0-4dd4-99c3-2509ba36b668","Type":"ContainerStarted","Data":"6f6dd2492439506c9819a2ed2520929ae7c3178e99bf5bfc5b56cd8dff324e7a"}
Oct 31 19:16:02.406314 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:02.406184    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv" podStartSLOduration=7.620288657 podStartE2EDuration="23.406170679s" totalImagesPullingTime="15.785882022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:02.35581404 +0000 UTC m=+23.919284372" watchObservedRunningTime="2026-10-31 19:16:02.406170679 +0000 UTC m=+23.969641009"
Oct 31 19:16:02.942733 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:02.942700    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:02.942941 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:02.942740    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:02.942941 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:02.942710    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:02.942941 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:02.942824    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:16:02.943102 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:02.942925    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:16:02.943102 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:02.943007    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:16:03.139630 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:03.139591    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:03.140050 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:03.139690    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:03.140050 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:03.139735    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:03.155944 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:03.155913    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:03.156115 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:03.156104    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:04.331428 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:04.331392    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vs8zn"]
Oct 31 19:16:04.331899 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:04.331504    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:04.331899 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:04.331613    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:16:04.334345 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:04.334315    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5rjx"]
Oct 31 19:16:04.334489 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:04.334420    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:04.334534 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:04.334509    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:16:04.335398 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:04.335369    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nptg8"]
Oct 31 19:16:04.335568 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:04.335534    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:04.335761 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:04.335701    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:16:05.942489 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:05.942408    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:05.942489 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:05.942476    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:05.942898 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:05.942501    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:05.942898 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:05.942582    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:16:05.942898 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:05.942639    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:16:05.942898 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:05.942719    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:16:07.941890 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:07.941855    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:07.942306 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:07.941855    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:07.942306 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:07.941966    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:16:07.942306 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:07.942066    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:16:07.942306 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:07.941855    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:07.942306 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:07.942155    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:16:09.941877 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:09.941777    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:09.941877 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:09.941843    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:09.942358 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:09.941930    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:09.942358 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:09.941942    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5rjx" podUID="c70d21f3-352d-45fc-853c-a74f690d1898"
Oct 31 19:16:09.942358 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:09.942017    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8zn" podUID="0c39943b-0a43-4184-b616-dda8d9f8545e"
Oct 31 19:16:09.942358 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:09.942138    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nptg8" podUID="1297e6ef-5035-497c-9777-c6b4448bceba"
Oct 31 19:16:10.215064 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.214980    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeReady"
Oct 31 19:16:10.215220 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.215144    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 19:16:10.326628 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.326590    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vmhw6"]
Oct 31 19:16:10.332927 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.332879    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"]
Oct 31 19:16:10.333310 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.333284    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vmhw6"
Oct 31 19:16:10.335624 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.335603    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gdrr2"]
Oct 31 19:16:10.335756 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.335725    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:10.337786 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.337767    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wq4hz"]
Oct 31 19:16:10.337918 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.337845    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.339806 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.339784    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"]
Oct 31 19:16:10.339908 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.339893    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wq4hz"
Oct 31 19:16:10.341802 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.341783    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mjspt"]
Oct 31 19:16:10.341916 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.341901    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:10.343838 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.343819    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"]
Oct 31 19:16:10.343968 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.343949    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mjspt"
Oct 31 19:16:10.346077 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.346057    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"]
Oct 31 19:16:10.346228 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.346156    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.348227 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.348212    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55c7865b76-fzmxv"]
Oct 31 19:16:10.348731 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.348714    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.350912 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.350888    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst"]
Oct 31 19:16:10.352439 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.352411    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.354132 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.354113    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nztjc"]
Oct 31 19:16:10.354232 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.354215    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst"
Oct 31 19:16:10.358229 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.358205    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.377293 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.377244    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65f799dcd5-gmv4v"]
Oct 31 19:16:10.382188 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.380037    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-ddc6dfd67-fk8lv"]
Oct 31 19:16:10.382188 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.381314    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.384234 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.384210    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg"]
Oct 31 19:16:10.384358 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.384326    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.385347 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.385322    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.390522 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.390497    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"]
Oct 31 19:16:10.390830 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.390809    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg"
Oct 31 19:16:10.391931 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.391906    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.395589 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.395565    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-62bwf"]
Oct 31 19:16:10.395839 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.395707    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:10.396393 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.396361    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 19:16:10.396393 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.396392    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rdjmx\""
Oct 31 19:16:10.401416 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.401394    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tcmmx"]
Oct 31 19:16:10.401560 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.401539    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:10.405244 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.405223    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vmhw6"]
Oct 31 19:16:10.405362 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.405256    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"]
Oct 31 19:16:10.405362 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.405272    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wq4hz"]
Oct 31 19:16:10.405362 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.405287    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"]
Oct 31 19:16:10.405362 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.405300    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gdrr2"]
Oct 31 19:16:10.405362 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.405345    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.430268 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430240    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg"]
Oct 31 19:16:10.430470 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430290    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.430603 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-trusted-ca\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.430603 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-bound-sa-token\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.430758 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-image-registry-private-configuration\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.430758 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430700    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.430758 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430749    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:10.430909 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430775    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdfnn\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-kube-api-access-fdfnn\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.430909 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430803    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-service-ca-bundle\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.430909 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430832    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae953c47-79ec-4415-9f33-d4fbb364d6c6-ca-trust-extracted\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.430909 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-certificates\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.430909 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430882    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-tmp\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.431158 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls8vr\" (UniqueName: \"kubernetes.io/projected/73377bc4-2624-4ed6-a5aa-30228b89cbfb-kube-api-access-ls8vr\") pod \"kas-connection-checker-685895c7c9-wq4hz\" (UID: \"73377bc4-2624-4ed6-a5aa-30228b89cbfb\") " pod="kube-system/kas-connection-checker-685895c7c9-wq4hz"
Oct 31 19:16:10.431158 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430953    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1b27776-d610-4efa-9f8e-e08f0161a07f-serving-cert\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.431158 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.430980    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1b27776-d610-4efa-9f8e-e08f0161a07f-config\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.431158 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38ce8722-6108-4502-b83d-0a29a46f88b0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:10.431158 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f59bf5f-da15-4174-a5be-b843478f0191-serving-cert\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.431158 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431060    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlfc\" (UniqueName: \"kubernetes.io/projected/b1e328bc-df61-43b5-a5cf-12a2389903a6-kube-api-access-wwlfc\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:10.431158 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431084    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9dh8\" (UniqueName: \"kubernetes.io/projected/78ed8e27-2cd2-43e5-8ee1-23042619743e-kube-api-access-v9dh8\") pod \"kas-connection-checker-685895c7c9-mjspt\" (UID: \"78ed8e27-2cd2-43e5-8ee1-23042619743e\") " pod="kube-system/kas-connection-checker-685895c7c9-mjspt"
Oct 31 19:16:10.431158 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431108    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-installation-pull-secrets\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.431158 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqdqq\" (UniqueName: \"kubernetes.io/projected/291cf47e-f759-4489-abb5-1bafffff7b25-kube-api-access-nqdqq\") pod \"kas-connection-checker-685895c7c9-vmhw6\" (UID: \"291cf47e-f759-4489-abb5-1bafffff7b25\") " pod="kube-system/kas-connection-checker-685895c7c9-vmhw6"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431232    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431262    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1e328bc-df61-43b5-a5cf-12a2389903a6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-config\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431343    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3897285-e4e6-45da-bf32-fcadd1f9b506-serving-cert\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqzf2\" (UniqueName: \"kubernetes.io/projected/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-kube-api-access-pqzf2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8qfj\" (UniqueName: \"kubernetes.io/projected/c0eaee38-fd4e-489a-a854-39245cd595dc-kube-api-access-p8qfj\") pod \"volume-data-source-validator-58c9f5f46d-24dst\" (UID: \"c0eaee38-fd4e-489a-a854-39245cd595dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-792p5\" (UniqueName: \"kubernetes.io/projected/1f59bf5f-da15-4174-a5be-b843478f0191-kube-api-access-792p5\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-trusted-ca\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-snapshots\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjmw8\" (UniqueName: \"kubernetes.io/projected/a3897285-e4e6-45da-bf32-fcadd1f9b506-kube-api-access-qjmw8\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.431613 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.431583    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prj9t\" (UniqueName: \"kubernetes.io/projected/c1b27776-d610-4efa-9f8e-e08f0161a07f-kube-api-access-prj9t\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.436740 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.436711    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst"]
Oct 31 19:16:10.436945 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.436928    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddc6dfd67-fk8lv"]
Oct 31 19:16:10.437460 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.437441    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55c7865b76-fzmxv"]
Oct 31 19:16:10.438871 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.438846    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65f799dcd5-gmv4v"]
Oct 31 19:16:10.442519 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.442491    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62bwf"]
Oct 31 19:16:10.447402 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.447296    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tcmmx"]
Oct 31 19:16:10.451014 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.450985    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"]
Oct 31 19:16:10.475910 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.475882    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mjspt"]
Oct 31 19:16:10.482806 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.482780    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"]
Oct 31 19:16:10.485974 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.485939    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.486145 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486074    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.486145 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486104    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 19:16:10.486145 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486128    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.486352 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486335    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486527    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486543    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486557    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486544    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486588    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486593    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486602    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486608    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486619    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486627    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486607    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486642    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486677    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 19:16:10.486697 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486691    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 19:16:10.487190 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486802    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.487190 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486894    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.487190 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486929    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 19:16:10.487190 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.486976    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 19:16:10.491161 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.491140    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 19:16:10.491161 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.491159    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.491328 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.491159    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 19:16:10.491328 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.491175    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 19:16:10.491328 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.491224    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 19:16:10.532004 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.531974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.532004 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1e328bc-df61-43b5-a5cf-12a2389903a6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:10.532212 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-config\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.532212 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532045    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:10.532212 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3897285-e4e6-45da-bf32-fcadd1f9b506-serving-cert\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.532212 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-trusted-ca\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.532212 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532125    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqzf2\" (UniqueName: \"kubernetes.io/projected/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-kube-api-access-pqzf2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.532385 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532209    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8qfj\" (UniqueName: \"kubernetes.io/projected/c0eaee38-fd4e-489a-a854-39245cd595dc-kube-api-access-p8qfj\") pod \"volume-data-source-validator-58c9f5f46d-24dst\" (UID: \"c0eaee38-fd4e-489a-a854-39245cd595dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst"
Oct 31 19:16:10.532385 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/425443c8-69c7-464e-9f85-0578ff4effdc-config-volume\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.532385 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-792p5\" (UniqueName: \"kubernetes.io/projected/1f59bf5f-da15-4174-a5be-b843478f0191-kube-api-access-792p5\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.532385 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-trusted-ca\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.532515 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532406    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.532515 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532440    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-bound-sa-token\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.532515 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532459    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50011131-90de-4df2-ba1e-28b2a919dbf0-cert\") pod \"ingress-canary-62bwf\" (UID: \"50011131-90de-4df2-ba1e-28b2a919dbf0\") " pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:10.532515 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532481    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-snapshots\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.532515 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532505    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmw8\" (UniqueName: \"kubernetes.io/projected/a3897285-e4e6-45da-bf32-fcadd1f9b506-kube-api-access-qjmw8\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.532700 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532539    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prj9t\" (UniqueName: \"kubernetes.io/projected/c1b27776-d610-4efa-9f8e-e08f0161a07f-kube-api-access-prj9t\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.532700 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532686    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9kt\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-kube-api-access-4b9kt\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.532770 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.532770 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-trusted-ca\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.532841 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79fc\" (UniqueName: \"kubernetes.io/projected/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-kube-api-access-v79fc\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.532902 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532883    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.532950 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532894    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-config\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.532950 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.533025 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.532982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-bound-sa-token\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.533067 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-image-registry-private-configuration\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.533067 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533050    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.533130 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533066    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.533130 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533079    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.533130 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-stats-auth\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.533228 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533142    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7f9m\" (UniqueName: \"kubernetes.io/projected/425443c8-69c7-464e-9f85-0578ff4effdc-kube-api-access-w7f9m\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.533228 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:10.533228 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533203    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-snapshots\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.533334 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533204    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdfnn\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-kube-api-access-fdfnn\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.533334 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533259    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-service-ca-bundle\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.533334 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.533271    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 19:16:10.533334 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533282    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae953c47-79ec-4415-9f33-d4fbb364d6c6-ca-trust-extracted\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.533334 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-installation-pull-secrets\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.533334 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.533325    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert podName:38ce8722-6108-4502-b83d-0a29a46f88b0 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:11.033312253 +0000 UTC m=+32.596782561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jcp2j" (UID: "38ce8722-6108-4502-b83d-0a29a46f88b0") : secret "networking-console-plugin-cert" not found
Oct 31 19:16:10.533634 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-default-certificate\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.533634 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-certificates\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.533634 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533416    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-tmp\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.533634 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533444    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-image-registry-private-configuration\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.533634 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74f1221c-9c54-4098-a301-ece68db990eb-ca-trust-extracted\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.533634 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533496    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:10.533634 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8p8\" (UniqueName: \"kubernetes.io/projected/ac149f85-b47b-4fb7-8ad7-3f985129f292-kube-api-access-7k8p8\") pod \"network-check-source-6f78d59f99-zkvsg\" (UID: \"ac149f85-b47b-4fb7-8ad7-3f985129f292\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg"
Oct 31 19:16:10.533634 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533579    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-registry-certificates\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.534031 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533661    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae953c47-79ec-4415-9f33-d4fbb364d6c6-ca-trust-extracted\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.534031 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533645    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls8vr\" (UniqueName: \"kubernetes.io/projected/73377bc4-2624-4ed6-a5aa-30228b89cbfb-kube-api-access-ls8vr\") pod \"kas-connection-checker-685895c7c9-wq4hz\" (UID: \"73377bc4-2624-4ed6-a5aa-30228b89cbfb\") " pod="kube-system/kas-connection-checker-685895c7c9-wq4hz"
Oct 31 19:16:10.534031 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533738    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-tmp\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.534031 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnkn\" (UniqueName: \"kubernetes.io/projected/50011131-90de-4df2-ba1e-28b2a919dbf0-kube-api-access-gtnkn\") pod \"ingress-canary-62bwf\" (UID: \"50011131-90de-4df2-ba1e-28b2a919dbf0\") " pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:10.534031 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533878    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1b27776-d610-4efa-9f8e-e08f0161a07f-serving-cert\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.534031 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533904    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1b27776-d610-4efa-9f8e-e08f0161a07f-config\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.534031 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533949    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-service-ca-bundle\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.534031 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.533947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.534031 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534001    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-certificates\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.534400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534068    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38ce8722-6108-4502-b83d-0a29a46f88b0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:10.534400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f59bf5f-da15-4174-a5be-b843478f0191-serving-cert\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.534400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534196    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlfc\" (UniqueName: \"kubernetes.io/projected/b1e328bc-df61-43b5-a5cf-12a2389903a6-kube-api-access-wwlfc\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:10.534400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfd9g\" (UniqueName: \"kubernetes.io/projected/25311b12-0544-48e2-8b00-f56aff90596d-kube-api-access-sfd9g\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:10.534400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dh8\" (UniqueName: \"kubernetes.io/projected/78ed8e27-2cd2-43e5-8ee1-23042619743e-kube-api-access-v9dh8\") pod \"kas-connection-checker-685895c7c9-mjspt\" (UID: \"78ed8e27-2cd2-43e5-8ee1-23042619743e\") " pod="kube-system/kas-connection-checker-685895c7c9-mjspt"
Oct 31 19:16:10.534400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-installation-pull-secrets\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.534400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534321    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/425443c8-69c7-464e-9f85-0578ff4effdc-tmp-dir\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.534400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534348    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdqq\" (UniqueName: \"kubernetes.io/projected/291cf47e-f759-4489-abb5-1bafffff7b25-kube-api-access-nqdqq\") pod \"kas-connection-checker-685895c7c9-vmhw6\" (UID: \"291cf47e-f759-4489-abb5-1bafffff7b25\") " pod="kube-system/kas-connection-checker-685895c7c9-vmhw6"
Oct 31 19:16:10.534400 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534372    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:10.534711 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534580    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1b27776-d610-4efa-9f8e-e08f0161a07f-config\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.534711 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.534690    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38ce8722-6108-4502-b83d-0a29a46f88b0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:10.536792 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.536769    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 19:16:10.543135 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.543104    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1e328bc-df61-43b5-a5cf-12a2389903a6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:10.582263 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.582227    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 19:16:10.582263 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.582244    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 19:16:10.582477 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.582434    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 19:16:10.582477 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.582444    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 19:16:10.582477 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.582466    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 19:16:10.582603 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.582482    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 19:16:10.583284 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.583263    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-trusted-ca\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.583429 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.583408    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"]
Oct 31 19:16:10.583853 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.583835    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.583925 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.583910    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-trusted-ca\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.585606 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.585591    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 19:16:10.585690 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.585670    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 19:16:10.585742 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.585676    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-db5zp\""
Oct 31 19:16:10.585742 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.585717    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgzhm\""
Oct 31 19:16:10.585742 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.585719    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-b72kc\""
Oct 31 19:16:10.585900 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.585785    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dhmj8\""
Oct 31 19:16:10.585900 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.585794    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 19:16:10.585900 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.585831    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8n9ml\""
Oct 31 19:16:10.585900 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.585896    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 19:16:10.586418 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586402    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 19:16:10.586511 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586491    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 19:16:10.586569 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586531    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 19:16:10.586569 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586502    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 19:16:10.586569 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586544    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ml5rl\""
Oct 31 19:16:10.586569 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586550    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 19:16:10.586569 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586554    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-s8g85\""
Oct 31 19:16:10.586835 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586584    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 19:16:10.586835 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586586    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 19:16:10.586835 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586615    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5n42z\""
Oct 31 19:16:10.586835 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586619    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sqb42\""
Oct 31 19:16:10.586835 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586718    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nztjc"]
Oct 31 19:16:10.586835 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586742    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-llq5c\""
Oct 31 19:16:10.586835 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586772    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mcs56\""
Oct 31 19:16:10.586835 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.586836    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t6kdz\""
Oct 31 19:16:10.587642 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.587623    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:10.592560 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.592530    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 19:16:10.592699 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.592624    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls podName:b1e328bc-df61-43b5-a5cf-12a2389903a6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:11.092601748 +0000 UTC m=+32.656072073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9pglj" (UID: "b1e328bc-df61-43b5-a5cf-12a2389903a6") : secret "cluster-monitoring-operator-tls" not found
Oct 31 19:16:10.593844 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.593820    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:10.593844 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.593843    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c7865b76-fzmxv: secret "image-registry-tls" not found
Oct 31 19:16:10.594114 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.594088    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls podName:ae953c47-79ec-4415-9f33-d4fbb364d6c6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:11.09406761 +0000 UTC m=+32.657537935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls") pod "image-registry-55c7865b76-fzmxv" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6") : secret "image-registry-tls" not found
Oct 31 19:16:10.594475 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.594284    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-bound-sa-token\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.595102 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.595082    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqzf2\" (UniqueName: \"kubernetes.io/projected/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-kube-api-access-pqzf2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.595187 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.595109    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dh8\" (UniqueName: \"kubernetes.io/projected/78ed8e27-2cd2-43e5-8ee1-23042619743e-kube-api-access-v9dh8\") pod \"kas-connection-checker-685895c7c9-mjspt\" (UID: \"78ed8e27-2cd2-43e5-8ee1-23042619743e\") " pod="kube-system/kas-connection-checker-685895c7c9-mjspt"
Oct 31 19:16:10.595336 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.595217    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8qfj\" (UniqueName: \"kubernetes.io/projected/c0eaee38-fd4e-489a-a854-39245cd595dc-kube-api-access-p8qfj\") pod \"volume-data-source-validator-58c9f5f46d-24dst\" (UID: \"c0eaee38-fd4e-489a-a854-39245cd595dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst"
Oct 31 19:16:10.595485 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.595465    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prj9t\" (UniqueName: \"kubernetes.io/projected/c1b27776-d610-4efa-9f8e-e08f0161a07f-kube-api-access-prj9t\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.595701 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.595637    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdqq\" (UniqueName: \"kubernetes.io/projected/291cf47e-f759-4489-abb5-1bafffff7b25-kube-api-access-nqdqq\") pod \"kas-connection-checker-685895c7c9-vmhw6\" (UID: \"291cf47e-f759-4489-abb5-1bafffff7b25\") " pod="kube-system/kas-connection-checker-685895c7c9-vmhw6"
Oct 31 19:16:10.596203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.596178    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdfnn\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-kube-api-access-fdfnn\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.596203 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.596194    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmw8\" (UniqueName: \"kubernetes.io/projected/a3897285-e4e6-45da-bf32-fcadd1f9b506-kube-api-access-qjmw8\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.596725 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.596699    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls8vr\" (UniqueName: \"kubernetes.io/projected/73377bc4-2624-4ed6-a5aa-30228b89cbfb-kube-api-access-ls8vr\") pod \"kas-connection-checker-685895c7c9-wq4hz\" (UID: \"73377bc4-2624-4ed6-a5aa-30228b89cbfb\") " pod="kube-system/kas-connection-checker-685895c7c9-wq4hz"
Oct 31 19:16:10.597342 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.597319    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1b27776-d610-4efa-9f8e-e08f0161a07f-serving-cert\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.597726 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.597710    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.597804 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.597733    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-image-registry-private-configuration\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.598229 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.598205    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f59bf5f-da15-4174-a5be-b843478f0191-serving-cert\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.598385 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.598366    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3897285-e4e6-45da-bf32-fcadd1f9b506-serving-cert\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.598502 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.598487    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-installation-pull-secrets\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:10.598560 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.598510    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlfc\" (UniqueName: \"kubernetes.io/projected/b1e328bc-df61-43b5-a5cf-12a2389903a6-kube-api-access-wwlfc\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:10.598699 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.598636    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-792p5\" (UniqueName: \"kubernetes.io/projected/1f59bf5f-da15-4174-a5be-b843478f0191-kube-api-access-792p5\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.635451 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635410    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-trusted-ca\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.635612 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635457    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/425443c8-69c7-464e-9f85-0578ff4effdc-config-volume\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.635612 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-bound-sa-token\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.635612 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50011131-90de-4df2-ba1e-28b2a919dbf0-cert\") pod \"ingress-canary-62bwf\" (UID: \"50011131-90de-4df2-ba1e-28b2a919dbf0\") " pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:10.635612 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635567    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9kt\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-kube-api-access-4b9kt\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.635864 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635699    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79fc\" (UniqueName: \"kubernetes.io/projected/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-kube-api-access-v79fc\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.635864 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.635864 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.635864 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.636049 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635876    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-stats-auth\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.636049 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7f9m\" (UniqueName: \"kubernetes.io/projected/425443c8-69c7-464e-9f85-0578ff4effdc-kube-api-access-w7f9m\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.636049 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.635926    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:10.636049 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.635944    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f799dcd5-gmv4v: secret "image-registry-tls" not found
Oct 31 19:16:10.636049 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635970    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-installation-pull-secrets\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.636049 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.635997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-default-certificate\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.636049 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.636038    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls podName:74f1221c-9c54-4098-a301-ece68db990eb nodeName:}" failed. No retries permitted until 2026-10-31 19:16:11.136020903 +0000 UTC m=+32.699491226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls") pod "image-registry-65f799dcd5-gmv4v" (UID: "74f1221c-9c54-4098-a301-ece68db990eb") : secret "image-registry-tls" not found
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.636074    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.636103    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:11.13608648 +0000 UTC m=+32.699556806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : configmap references non-existent config key: service-ca.crt
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.636134    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:11.136119488 +0000 UTC m=+32.699589806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : secret "router-metrics-certs-default" not found
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-image-registry-private-configuration\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636186    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74f1221c-9c54-4098-a301-ece68db990eb-ca-trust-extracted\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636185    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/425443c8-69c7-464e-9f85-0578ff4effdc-config-volume\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636248    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8p8\" (UniqueName: \"kubernetes.io/projected/ac149f85-b47b-4fb7-8ad7-3f985129f292-kube-api-access-7k8p8\") pod \"network-check-source-6f78d59f99-zkvsg\" (UID: \"ac149f85-b47b-4fb7-8ad7-3f985129f292\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg"
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636280    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-registry-certificates\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636315    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnkn\" (UniqueName: \"kubernetes.io/projected/50011131-90de-4df2-ba1e-28b2a919dbf0-kube-api-access-gtnkn\") pod \"ingress-canary-62bwf\" (UID: \"50011131-90de-4df2-ba1e-28b2a919dbf0\") " pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.636388 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfd9g\" (UniqueName: \"kubernetes.io/projected/25311b12-0544-48e2-8b00-f56aff90596d-kube-api-access-sfd9g\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:10.637047 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636453    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/425443c8-69c7-464e-9f85-0578ff4effdc-tmp-dir\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.637047 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636476    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-trusted-ca\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.637047 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636485    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:10.637047 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636766    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74f1221c-9c54-4098-a301-ece68db990eb-ca-trust-extracted\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.637047 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.636845    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 19:16:10.637047 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.636900    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls podName:25311b12-0544-48e2-8b00-f56aff90596d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:11.13688679 +0000 UTC m=+32.700357118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4d9vb" (UID: "25311b12-0544-48e2-8b00-f56aff90596d") : secret "samples-operator-tls" not found
Oct 31 19:16:10.637047 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.636907    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/425443c8-69c7-464e-9f85-0578ff4effdc-tmp-dir\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.637379 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.637176    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 19:16:10.637379 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:10.637230    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls podName:425443c8-69c7-464e-9f85-0578ff4effdc nodeName:}" failed. No retries permitted until 2026-10-31 19:16:11.137215939 +0000 UTC m=+32.700686270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls") pod "dns-default-tcmmx" (UID: "425443c8-69c7-464e-9f85-0578ff4effdc") : secret "dns-default-metrics-tls" not found
Oct 31 19:16:10.637559 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.637537    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-registry-certificates\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.637640 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.637561    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:10.638766 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.638736    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50011131-90de-4df2-ba1e-28b2a919dbf0-cert\") pod \"ingress-canary-62bwf\" (UID: \"50011131-90de-4df2-ba1e-28b2a919dbf0\") " pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:10.638866 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.638788    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-default-certificate\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.638927 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.638908    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-installation-pull-secrets\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.638995 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.638977    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-image-registry-private-configuration\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.639161 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.639146    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-stats-auth\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.647572 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.647542    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vmhw6"
Oct 31 19:16:10.663635 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.663606    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:10.670781 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.670753    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wq4hz"
Oct 31 19:16:10.686616 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.686582    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mjspt"
Oct 31 19:16:10.697201 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.697159    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:10.698168 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.697671    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9kt\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-kube-api-access-4b9kt\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.698168 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.698124    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8p8\" (UniqueName: \"kubernetes.io/projected/ac149f85-b47b-4fb7-8ad7-3f985129f292-kube-api-access-7k8p8\") pod \"network-check-source-6f78d59f99-zkvsg\" (UID: \"ac149f85-b47b-4fb7-8ad7-3f985129f292\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg"
Oct 31 19:16:10.698577 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.698543    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7f9m\" (UniqueName: \"kubernetes.io/projected/425443c8-69c7-464e-9f85-0578ff4effdc-kube-api-access-w7f9m\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:10.699937 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.698934    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-bound-sa-token\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:10.699937 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.699188    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfd9g\" (UniqueName: \"kubernetes.io/projected/25311b12-0544-48e2-8b00-f56aff90596d-kube-api-access-sfd9g\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:10.699937 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.699421    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnkn\" (UniqueName: \"kubernetes.io/projected/50011131-90de-4df2-ba1e-28b2a919dbf0-kube-api-access-gtnkn\") pod \"ingress-canary-62bwf\" (UID: \"50011131-90de-4df2-ba1e-28b2a919dbf0\") " pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:10.701076 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.701053    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79fc\" (UniqueName: \"kubernetes.io/projected/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-kube-api-access-v79fc\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:10.704101 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.704076    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:10.716610 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.716573    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst"
Oct 31 19:16:10.724094 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.724072    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:10.757205 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.757084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg"
Oct 31 19:16:10.770835 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.770794    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:10.865884 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:10.865848    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:11.041685 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.041629    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:11.042098 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.041757    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 19:16:11.042098 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.041827    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert podName:38ce8722-6108-4502-b83d-0a29a46f88b0 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.041810799 +0000 UTC m=+33.605281108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jcp2j" (UID: "38ce8722-6108-4502-b83d-0a29a46f88b0") : secret "networking-console-plugin-cert" not found
Oct 31 19:16:11.118045 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.118016    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vmhw6"]
Oct 31 19:16:11.119359 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:11.119328    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291cf47e_f759_4489_abb5_1bafffff7b25.slice/crio-a8947f9b17cb096a6d5fb051ce149de3bbd1a6efa6d854d62591418947562cc9 WatchSource:0}: Error finding container a8947f9b17cb096a6d5fb051ce149de3bbd1a6efa6d854d62591418947562cc9: Status 404 returned error can't find the container with id a8947f9b17cb096a6d5fb051ce149de3bbd1a6efa6d854d62591418947562cc9
Oct 31 19:16:11.124454 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:11.124426    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f59bf5f_da15_4174_a5be_b843478f0191.slice/crio-ca557849d78f9ca1965ca46ec0e7e6d7a7174dcdd63d56f448038f9d1e1951e8 WatchSource:0}: Error finding container ca557849d78f9ca1965ca46ec0e7e6d7a7174dcdd63d56f448038f9d1e1951e8: Status 404 returned error can't find the container with id ca557849d78f9ca1965ca46ec0e7e6d7a7174dcdd63d56f448038f9d1e1951e8
Oct 31 19:16:11.125811 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.125788    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gdrr2"]
Oct 31 19:16:11.126556 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.126542    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst"]
Oct 31 19:16:11.128180 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:11.128158    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0eaee38_fd4e_489a_a854_39245cd595dc.slice/crio-facc7d3c43fa65af9418b01f85949f531d104e9625d6ddc915228cbcf4362306 WatchSource:0}: Error finding container facc7d3c43fa65af9418b01f85949f531d104e9625d6ddc915228cbcf4362306: Status 404 returned error can't find the container with id facc7d3c43fa65af9418b01f85949f531d104e9625d6ddc915228cbcf4362306
Oct 31 19:16:11.142816 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.142775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:11.143000 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.142835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:11.143000 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.142905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:11.143000 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.142934    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.142912135 +0000 UTC m=+33.706382462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : configmap references non-existent config key: service-ca.crt
Oct 31 19:16:11.143000 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.142983    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 19:16:11.143000 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.142994    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:11.143237 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143039    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls podName:25311b12-0544-48e2-8b00-f56aff90596d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.143022364 +0000 UTC m=+33.706492692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4d9vb" (UID: "25311b12-0544-48e2-8b00-f56aff90596d") : secret "samples-operator-tls" not found
Oct 31 19:16:11.143237 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.142984    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:11.143237 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.143070    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:11.143237 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143077    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c7865b76-fzmxv: secret "image-registry-tls" not found
Oct 31 19:16:11.143237 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.143175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:11.143237 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143201    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls podName:ae953c47-79ec-4415-9f33-d4fbb364d6c6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.143189609 +0000 UTC m=+33.706659948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls") pod "image-registry-55c7865b76-fzmxv" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6") : secret "image-registry-tls" not found
Oct 31 19:16:11.143237 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.143219    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:11.143237 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143229    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 19:16:11.143612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143255    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.143248179 +0000 UTC m=+33.706718534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : secret "router-metrics-certs-default" not found
Oct 31 19:16:11.143612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143083    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 19:16:11.143612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143277    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls podName:425443c8-69c7-464e-9f85-0578ff4effdc nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.143272018 +0000 UTC m=+33.706742392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls") pod "dns-default-tcmmx" (UID: "425443c8-69c7-464e-9f85-0578ff4effdc") : secret "dns-default-metrics-tls" not found
Oct 31 19:16:11.143612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143138    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 19:16:11.143612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143299    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls podName:b1e328bc-df61-43b5-a5cf-12a2389903a6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.143293148 +0000 UTC m=+33.706763458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9pglj" (UID: "b1e328bc-df61-43b5-a5cf-12a2389903a6") : secret "cluster-monitoring-operator-tls" not found
Oct 31 19:16:11.143612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143301    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:11.143612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143309    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f799dcd5-gmv4v: secret "image-registry-tls" not found
Oct 31 19:16:11.143612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:11.143344    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls podName:74f1221c-9c54-4098-a301-ece68db990eb nodeName:}" failed. No retries permitted until 2026-10-31 19:16:12.143332862 +0000 UTC m=+33.706803184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls") pod "image-registry-65f799dcd5-gmv4v" (UID: "74f1221c-9c54-4098-a301-ece68db990eb") : secret "image-registry-tls" not found
Oct 31 19:16:11.163412 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.163379    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vmhw6" event={"ID":"291cf47e-f759-4489-abb5-1bafffff7b25","Type":"ContainerStarted","Data":"a8947f9b17cb096a6d5fb051ce149de3bbd1a6efa6d854d62591418947562cc9"}
Oct 31 19:16:11.165162 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.165086    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst" event={"ID":"c0eaee38-fd4e-489a-a854-39245cd595dc","Type":"ContainerStarted","Data":"facc7d3c43fa65af9418b01f85949f531d104e9625d6ddc915228cbcf4362306"}
Oct 31 19:16:11.165301 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.165193    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mjspt"]
Oct 31 19:16:11.165934 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:11.165904    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ed8e27_2cd2_43e5_8ee1_23042619743e.slice/crio-f2e931f8dde77690c89d31097dc30f8fc188ce421aa57ca6e2d03bf3e6611313 WatchSource:0}: Error finding container f2e931f8dde77690c89d31097dc30f8fc188ce421aa57ca6e2d03bf3e6611313: Status 404 returned error can't find the container with id f2e931f8dde77690c89d31097dc30f8fc188ce421aa57ca6e2d03bf3e6611313
Oct 31 19:16:11.166768 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.166739    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gdrr2" event={"ID":"1f59bf5f-da15-4174-a5be-b843478f0191","Type":"ContainerStarted","Data":"ca557849d78f9ca1965ca46ec0e7e6d7a7174dcdd63d56f448038f9d1e1951e8"}
Oct 31 19:16:11.192500 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:11.192453    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73377bc4_2624_4ed6_a5aa_30228b89cbfb.slice/crio-f7ec2ee3429cb367b4b4cfef810d172f37250b5cdd37c27e0a09d54ad6dd104e WatchSource:0}: Error finding container f7ec2ee3429cb367b4b4cfef810d172f37250b5cdd37c27e0a09d54ad6dd104e: Status 404 returned error can't find the container with id f7ec2ee3429cb367b4b4cfef810d172f37250b5cdd37c27e0a09d54ad6dd104e
Oct 31 19:16:11.192688 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.192615    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wq4hz"]
Oct 31 19:16:11.227320 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.227279    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"]
Oct 31 19:16:11.227777 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:11.227731    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38ebb5a_bd63_4e8c_94df_c0a904503c4c.slice/crio-fc1ea5cc45e4f1d27a8ea54a3a33042f9120ccb107860f4dbc52028e9fe25f39 WatchSource:0}: Error finding container fc1ea5cc45e4f1d27a8ea54a3a33042f9120ccb107860f4dbc52028e9fe25f39: Status 404 returned error can't find the container with id fc1ea5cc45e4f1d27a8ea54a3a33042f9120ccb107860f4dbc52028e9fe25f39
Oct 31 19:16:11.254099 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.254069    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62bwf"]
Oct 31 19:16:11.256957 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:11.256925    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50011131_90de_4df2_ba1e_28b2a919dbf0.slice/crio-dabb61cab0ada53f5bdc5279952acc1249bacd3595d93d0fa33e7ba42dfa85d7 WatchSource:0}: Error finding container dabb61cab0ada53f5bdc5279952acc1249bacd3595d93d0fa33e7ba42dfa85d7: Status 404 returned error can't find the container with id dabb61cab0ada53f5bdc5279952acc1249bacd3595d93d0fa33e7ba42dfa85d7
Oct 31 19:16:11.263620 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.263590    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nztjc"]
Oct 31 19:16:11.268153 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.268124    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg"]
Oct 31 19:16:11.276309 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:11.276281    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac149f85_b47b_4fb7_8ad7_3f985129f292.slice/crio-cbf00387d7f8b5f214ba89e3e762a65b0ad63bb748bd0d21c0b56eb9d6da04ea WatchSource:0}: Error finding container cbf00387d7f8b5f214ba89e3e762a65b0ad63bb748bd0d21c0b56eb9d6da04ea: Status 404 returned error can't find the container with id cbf00387d7f8b5f214ba89e3e762a65b0ad63bb748bd0d21c0b56eb9d6da04ea
Oct 31 19:16:11.286886 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.286857    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"]
Oct 31 19:16:11.297862 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:11.297834    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b27776_d610_4efa_9f8e_e08f0161a07f.slice/crio-82dcbd39451e591ca43cf604e1f62e55fef66d5f326889e7867d1447a7ee2a93 WatchSource:0}: Error finding container 82dcbd39451e591ca43cf604e1f62e55fef66d5f326889e7867d1447a7ee2a93: Status 404 returned error can't find the container with id 82dcbd39451e591ca43cf604e1f62e55fef66d5f326889e7867d1447a7ee2a93
Oct 31 19:16:11.942131 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.942091    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:11.942294 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.942175    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:11.942294 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.942193    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:11.995190 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.995148    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 19:16:11.996146 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.995779    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgqq8\""
Oct 31 19:16:11.996146 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.995805    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tgkwh\""
Oct 31 19:16:11.996146 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.996062    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qh5p\""
Oct 31 19:16:11.996351 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:11.996318    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 19:16:12.050452 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.050413    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:12.050854 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.050590    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 19:16:12.050854 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.050688    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert podName:38ce8722-6108-4502-b83d-0a29a46f88b0 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.050670574 +0000 UTC m=+35.614140901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jcp2j" (UID: "38ce8722-6108-4502-b83d-0a29a46f88b0") : secret "networking-console-plugin-cert" not found
Oct 31 19:16:12.151942 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.151905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:12.151942 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.151952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:12.152184 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.151976    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:12.152184 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.151994    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:12.152184 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152074    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:12.152184 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152089    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c7865b76-fzmxv: secret "image-registry-tls" not found
Oct 31 19:16:12.152184 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152119    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.152100919 +0000 UTC m=+35.715571242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : configmap references non-existent config key: service-ca.crt
Oct 31 19:16:12.152184 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152074    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 19:16:12.152184 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152144    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls podName:ae953c47-79ec-4415-9f33-d4fbb364d6c6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.15213639 +0000 UTC m=+35.715606698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls") pod "image-registry-55c7865b76-fzmxv" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6") : secret "image-registry-tls" not found
Oct 31 19:16:12.152184 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152168    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.152153853 +0000 UTC m=+35.715624208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : secret "router-metrics-certs-default" not found
Oct 31 19:16:12.152184 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152172    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:12.152184 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152186    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f799dcd5-gmv4v: secret "image-registry-tls" not found
Oct 31 19:16:12.152612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152239    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls podName:74f1221c-9c54-4098-a301-ece68db990eb nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.152225055 +0000 UTC m=+35.715695379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls") pod "image-registry-65f799dcd5-gmv4v" (UID: "74f1221c-9c54-4098-a301-ece68db990eb") : secret "image-registry-tls" not found
Oct 31 19:16:12.152612 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.152277    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:12.152612 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.152322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:12.152612 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.152376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:12.152612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152410    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 19:16:12.152612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152457    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 19:16:12.152612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152478    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls podName:25311b12-0544-48e2-8b00-f56aff90596d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.152462044 +0000 UTC m=+35.715932375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4d9vb" (UID: "25311b12-0544-48e2-8b00-f56aff90596d") : secret "samples-operator-tls" not found
Oct 31 19:16:12.152612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152499    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls podName:425443c8-69c7-464e-9f85-0578ff4effdc nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.152488729 +0000 UTC m=+35.715959044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls") pod "dns-default-tcmmx" (UID: "425443c8-69c7-464e-9f85-0578ff4effdc") : secret "dns-default-metrics-tls" not found
Oct 31 19:16:12.152612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152517    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 19:16:12.152612 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.152561    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls podName:b1e328bc-df61-43b5-a5cf-12a2389903a6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.152548968 +0000 UTC m=+35.716019305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9pglj" (UID: "b1e328bc-df61-43b5-a5cf-12a2389903a6") : secret "cluster-monitoring-operator-tls" not found
Oct 31 19:16:12.170246 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.170207    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l" event={"ID":"c1b27776-d610-4efa-9f8e-e08f0161a07f","Type":"ContainerStarted","Data":"82dcbd39451e591ca43cf604e1f62e55fef66d5f326889e7867d1447a7ee2a93"}
Oct 31 19:16:12.171831 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.171799    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wq4hz" event={"ID":"73377bc4-2624-4ed6-a5aa-30228b89cbfb","Type":"ContainerStarted","Data":"c8f7a8c6bc2872cc482779208cef3af34dfe906f8e58bc8b66322ece567ba410"}
Oct 31 19:16:12.171982 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.171856    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wq4hz" event={"ID":"73377bc4-2624-4ed6-a5aa-30228b89cbfb","Type":"ContainerStarted","Data":"f7ec2ee3429cb367b4b4cfef810d172f37250b5cdd37c27e0a09d54ad6dd104e"}
Oct 31 19:16:12.173023 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.172980    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg" event={"ID":"ac149f85-b47b-4fb7-8ad7-3f985129f292","Type":"ContainerStarted","Data":"cbf00387d7f8b5f214ba89e3e762a65b0ad63bb748bd0d21c0b56eb9d6da04ea"}
Oct 31 19:16:12.174178 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.174154    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62bwf" event={"ID":"50011131-90de-4df2-ba1e-28b2a919dbf0","Type":"ContainerStarted","Data":"dabb61cab0ada53f5bdc5279952acc1249bacd3595d93d0fa33e7ba42dfa85d7"}
Oct 31 19:16:12.175340 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.175277    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82" event={"ID":"b38ebb5a-bd63-4e8c-94df-c0a904503c4c","Type":"ContainerStarted","Data":"fc1ea5cc45e4f1d27a8ea54a3a33042f9120ccb107860f4dbc52028e9fe25f39"}
Oct 31 19:16:12.176766 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.176741    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vmhw6" event={"ID":"291cf47e-f759-4489-abb5-1bafffff7b25","Type":"ContainerStarted","Data":"78f17b872d656d0ea45f195ccb51fd39205d7ab666f4066f4bfab82f98fcd9bf"}
Oct 31 19:16:12.177952 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.177930    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" event={"ID":"a3897285-e4e6-45da-bf32-fcadd1f9b506","Type":"ContainerStarted","Data":"8cb1a8dd7005964caeb03488c7f4b3620912c1271d6c2db36698fa0c378bd691"}
Oct 31 19:16:12.179416 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.179384    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mjspt" event={"ID":"78ed8e27-2cd2-43e5-8ee1-23042619743e","Type":"ContainerStarted","Data":"9fe69d45881fc3cc7d377d2e21a4c764127f025ec012ff7b336b02e07ee5660e"}
Oct 31 19:16:12.179529 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.179422    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mjspt" event={"ID":"78ed8e27-2cd2-43e5-8ee1-23042619743e","Type":"ContainerStarted","Data":"f2e931f8dde77690c89d31097dc30f8fc188ce421aa57ca6e2d03bf3e6611313"}
Oct 31 19:16:12.480558 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.480116    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mjspt" podStartSLOduration=432.480098553 podStartE2EDuration="7m12.480098553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:12.478522787 +0000 UTC m=+34.041993117" watchObservedRunningTime="2026-10-31 19:16:12.480098553 +0000 UTC m=+34.043568883"
Oct 31 19:16:12.592309 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.591991    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vmhw6" podStartSLOduration=432.591971451 podStartE2EDuration="7m12.591971451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:12.590713948 +0000 UTC m=+34.154184280" watchObservedRunningTime="2026-10-31 19:16:12.591971451 +0000 UTC m=+34.155441783"
Oct 31 19:16:12.703888 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.703823    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wq4hz" podStartSLOduration=432.703800413 podStartE2EDuration="7m12.703800413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:12.702898038 +0000 UTC m=+34.266368371" watchObservedRunningTime="2026-10-31 19:16:12.703800413 +0000 UTC m=+34.267270743"
Oct 31 19:16:12.777281 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.777172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:12.777281 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.777247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:12.777888 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.777865    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 19:16:12.777983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:12.777943    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs podName:1297e6ef-5035-497c-9777-c6b4448bceba nodeName:}" failed. No retries permitted until 2026-10-31 19:16:44.777924589 +0000 UTC m=+66.341394913 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs") pod "network-metrics-daemon-nptg8" (UID: "1297e6ef-5035-497c-9777-c6b4448bceba") : secret "metrics-daemon-secret" not found
Oct 31 19:16:12.804678 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.800524    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:12.878308 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.878263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:12.881496 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.881463    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:12.903162 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.903122    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:12.925318 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:12.925276    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:13.346227 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:13.346093    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5rjx"]
Oct 31 19:16:13.347186 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:13.347106    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70d21f3_352d_45fc_853c_a74f690d1898.slice/crio-bbc6dc8dea02d76785975ab983ade811897aac5fc2b9b2a2727b82e28ec2f61f WatchSource:0}: Error finding container bbc6dc8dea02d76785975ab983ade811897aac5fc2b9b2a2727b82e28ec2f61f: Status 404 returned error can't find the container with id bbc6dc8dea02d76785975ab983ade811897aac5fc2b9b2a2727b82e28ec2f61f
Oct 31 19:16:13.378706 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:13.378390    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vs8zn"]
Oct 31 19:16:13.384602 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:13.384292    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c39943b_0a43_4184_b616_dda8d9f8545e.slice/crio-ce53bbfb2bf87bb9146509806c20754fffd7ff4bf7ca58db50ee758ac749a479 WatchSource:0}: Error finding container ce53bbfb2bf87bb9146509806c20754fffd7ff4bf7ca58db50ee758ac749a479: Status 404 returned error can't find the container with id ce53bbfb2bf87bb9146509806c20754fffd7ff4bf7ca58db50ee758ac749a479
Oct 31 19:16:14.093961 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.093743    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:14.093961 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.093900    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 19:16:14.094198 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.093980    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert podName:38ce8722-6108-4502-b83d-0a29a46f88b0 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:18.093959501 +0000 UTC m=+39.657429812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jcp2j" (UID: "38ce8722-6108-4502-b83d-0a29a46f88b0") : secret "networking-console-plugin-cert" not found
Oct 31 19:16:14.192750 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.192635    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5rjx" event={"ID":"c70d21f3-352d-45fc-853c-a74f690d1898","Type":"ContainerStarted","Data":"bbc6dc8dea02d76785975ab983ade811897aac5fc2b9b2a2727b82e28ec2f61f"}
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.194338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.194387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194506    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194524    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f799dcd5-gmv4v: secret "image-registry-tls" not found
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.194525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.194558    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194581    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls podName:74f1221c-9c54-4098-a301-ece68db990eb nodeName:}" failed. No retries permitted until 2026-10-31 19:16:18.194562859 +0000 UTC m=+39.758033174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls") pod "image-registry-65f799dcd5-gmv4v" (UID: "74f1221c-9c54-4098-a301-ece68db990eb") : secret "image-registry-tls" not found
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.194662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194671    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194677    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:18.194659893 +0000 UTC m=+39.758130217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : configmap references non-existent config key: service-ca.crt
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194733    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.194745    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194771    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls podName:25311b12-0544-48e2-8b00-f56aff90596d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:18.194760042 +0000 UTC m=+39.758230358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4d9vb" (UID: "25311b12-0544-48e2-8b00-f56aff90596d") : secret "samples-operator-tls" not found
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194820    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.194825    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194859    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls podName:425443c8-69c7-464e-9f85-0578ff4effdc nodeName:}" failed. No retries permitted until 2026-10-31 19:16:18.194849264 +0000 UTC m=+39.758319575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls") pod "dns-default-tcmmx" (UID: "425443c8-69c7-464e-9f85-0578ff4effdc") : secret "dns-default-metrics-tls" not found
Oct 31 19:16:14.194983 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194902    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 19:16:14.195941 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194941    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls podName:b1e328bc-df61-43b5-a5cf-12a2389903a6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:18.194928747 +0000 UTC m=+39.758399069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9pglj" (UID: "b1e328bc-df61-43b5-a5cf-12a2389903a6") : secret "cluster-monitoring-operator-tls" not found
Oct 31 19:16:14.195941 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194682    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c7865b76-fzmxv: secret "image-registry-tls" not found
Oct 31 19:16:14.195941 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194979    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls podName:ae953c47-79ec-4415-9f33-d4fbb364d6c6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:18.194968378 +0000 UTC m=+39.758438687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls") pod "image-registry-55c7865b76-fzmxv" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6") : secret "image-registry-tls" not found
Oct 31 19:16:14.195941 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.194504    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 19:16:14.195941 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:14.195018    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:18.195009943 +0000 UTC m=+39.758480294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : secret "router-metrics-certs-default" not found
Oct 31 19:16:14.199102 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.199047    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst" event={"ID":"c0eaee38-fd4e-489a-a854-39245cd595dc","Type":"ContainerStarted","Data":"b30b0109d2246837129bf95f81e83807283d33a240a72da73959652528cced3c"}
Oct 31 19:16:14.202310 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.202277    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vs8zn" event={"ID":"0c39943b-0a43-4184-b616-dda8d9f8545e","Type":"ContainerStarted","Data":"ce53bbfb2bf87bb9146509806c20754fffd7ff4bf7ca58db50ee758ac749a479"}
Oct 31 19:16:14.310494 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.310439    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst" podStartSLOduration=405.694359157 podStartE2EDuration="6m48.310419044s" totalImagesPullingTime="2.616059887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:14.309934643 +0000 UTC m=+35.873404997" watchObservedRunningTime="2026-10-31 19:16:14.310419044 +0000 UTC m=+35.873889377"
Oct 31 19:16:14.501066 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:14.500978    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:15.064990 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:15.064914    2574 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage386795343/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T191613Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=c1e75bec5702707d3aa343eb516bd6ba4852fe8fdf5e34e4718b9eb88c9c1098&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793475073~hmac=763be10cef5f96a8718ef78e3cd073acba07ee3636422a8530345521eb9101fc\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb08b4955abf8a0ffea0e4f2e446fbb6a5f3271312a996633901621b91b106ab"
Oct 31 19:16:15.065271 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:15.065225    2574 kuberuntime_manager.go:1707] "Unhandled Error" err="container serve-healthcheck-canary start failed in pod ingress-canary-62bwf_openshift-ingress-canary(50011131-90de-4df2-ba1e-28b2a919dbf0): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage386795343/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T191613Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=c1e75bec5702707d3aa343eb516bd6ba4852fe8fdf5e34e4718b9eb88c9c1098&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793475073~hmac=763be10cef5f96a8718ef78e3cd073acba07ee3636422a8530345521eb9101fc\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" logger="UnhandledError"
Oct 31 19:16:15.065366 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:15.065310    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"serve-healthcheck-canary\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage386795343/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T191613Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=c1e75bec5702707d3aa343eb516bd6ba4852fe8fdf5e34e4718b9eb88c9c1098&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793475073~hmac=763be10cef5f96a8718ef78e3cd073acba07ee3636422a8530345521eb9101fc\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress-canary/ingress-canary-62bwf" podUID="50011131-90de-4df2-ba1e-28b2a919dbf0"
Oct 31 19:16:15.208602 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:15.208419    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"serve-healthcheck-canary\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb08b4955abf8a0ffea0e4f2e446fbb6a5f3271312a996633901621b91b106ab\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage386795343/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T191613Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=c1e75bec5702707d3aa343eb516bd6ba4852fe8fdf5e34e4718b9eb88c9c1098&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793475073~hmac=763be10cef5f96a8718ef78e3cd073acba07ee3636422a8530345521eb9101fc\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress-canary/ingress-canary-62bwf" podUID="50011131-90de-4df2-ba1e-28b2a919dbf0"
Oct 31 19:16:16.212881 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:16.212786    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gdrr2" event={"ID":"1f59bf5f-da15-4174-a5be-b843478f0191","Type":"ContainerStarted","Data":"cb6021c50dd5a4a89277ea557aea8b80b4419b129f8e7e1089b9389447f68d22"}
Oct 31 19:16:16.375574 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:16.375510    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-gdrr2" podStartSLOduration=405.644612656 podStartE2EDuration="6m50.375487525s" totalImagesPullingTime="4.730874869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:16.325718026 +0000 UTC m=+37.889188359" watchObservedRunningTime="2026-10-31 19:16:16.375487525 +0000 UTC m=+37.938957857"
Oct 31 19:16:17.218639 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.218605    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/0.log"
Oct 31 19:16:17.219137 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.218674    2574 generic.go:356] "Generic (PLEG): container finished" podID="a3897285-e4e6-45da-bf32-fcadd1f9b506" containerID="6acf9bd878fef1281d9d09dab4bf8eb21b2acf31623ace09072c3d39ddf2496e" exitCode=255
Oct 31 19:16:17.219137 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.218765    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" event={"ID":"a3897285-e4e6-45da-bf32-fcadd1f9b506","Type":"ContainerDied","Data":"6acf9bd878fef1281d9d09dab4bf8eb21b2acf31623ace09072c3d39ddf2496e"}
Oct 31 19:16:17.219137 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.219023    2574 scope.go:122] "RemoveContainer" containerID="6acf9bd878fef1281d9d09dab4bf8eb21b2acf31623ace09072c3d39ddf2496e"
Oct 31 19:16:17.221308 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.220829    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l" event={"ID":"c1b27776-d610-4efa-9f8e-e08f0161a07f","Type":"ContainerStarted","Data":"a81491b9148450764b17496eefc78e77a22ce716b3e7b8391db80237cfed7974"}
Oct 31 19:16:17.348752 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.347488    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fnscp"]
Oct 31 19:16:17.374366 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.374291    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.427282 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.427250    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 19:16:17.427282 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.427256    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 19:16:17.427516 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.427316    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4b89l\""
Oct 31 19:16:17.434116 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.434077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-data-volume\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.434116 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.434116    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.434331 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.434187    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.434331 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.434253    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7sm\" (UniqueName: \"kubernetes.io/projected/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-api-access-8v7sm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.434430 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.434332    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-crio-socket\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.439257 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.439229    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fnscp"]
Oct 31 19:16:17.443340 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.443281    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l" podStartSLOduration=406.018993174 podStartE2EDuration="6m51.443263835s" totalImagesPullingTime="5.424270661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:17.442907048 +0000 UTC m=+39.006377379" watchObservedRunningTime="2026-10-31 19:16:17.443263835 +0000 UTC m=+39.006734167"
Oct 31 19:16:17.535430 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.535341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.535588 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:17.535509    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:17.535588 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:17.535587    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:18.035566614 +0000 UTC m=+39.599036938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:17.535724 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.535521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7sm\" (UniqueName: \"kubernetes.io/projected/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-api-access-8v7sm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.535832 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.535816    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-crio-socket\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.535885 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.535862    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-data-volume\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.535934 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.535889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.535998 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.535978    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-crio-socket\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.536057 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:17.536000    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 19:16:17.536057 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:17.536049    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:18.036033784 +0000 UTC m=+39.599504115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : configmap "kube-rbac-proxy" not found
Oct 31 19:16:17.536310 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.536294    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-data-volume\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.595051 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.595025    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7sm\" (UniqueName: \"kubernetes.io/projected/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-api-access-8v7sm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:17.602078 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.602044    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wl28r-zphd5"]
Oct 31 19:16:17.634518 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.634487    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.657794 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.657726    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wl28r-zphd5"]
Oct 31 19:16:17.663397 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.663363    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:17.745753 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.739758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1b0ae648-a85e-4a86-98f1-db2e9600c67a-archives-path\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.745753 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.740009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-service-ca-bundle\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.745753 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.740039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knvdl\" (UniqueName: \"kubernetes.io/projected/1b0ae648-a85e-4a86-98f1-db2e9600c67a-kube-api-access-knvdl\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.745753 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.740112    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-trusted-ca-bundle\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.840799 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.840758    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-service-ca-bundle\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.840799 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.840801    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knvdl\" (UniqueName: \"kubernetes.io/projected/1b0ae648-a85e-4a86-98f1-db2e9600c67a-kube-api-access-knvdl\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.841035 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.840848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-trusted-ca-bundle\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.841035 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.840882    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1b0ae648-a85e-4a86-98f1-db2e9600c67a-archives-path\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.841298 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.841283    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1b0ae648-a85e-4a86-98f1-db2e9600c67a-archives-path\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.841639 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.841625    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-service-ca-bundle\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.842539 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.842513    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-trusted-ca-bundle\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.898962 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.898927    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knvdl\" (UniqueName: \"kubernetes.io/projected/1b0ae648-a85e-4a86-98f1-db2e9600c67a-kube-api-access-knvdl\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:17.900051 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.900021    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:17.980512 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:17.980235    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:18.042914 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.042875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:18.043097 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.042971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:18.043097 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.042988    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 19:16:18.043097 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.043064    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:19.043043949 +0000 UTC m=+40.606514264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : configmap "kube-rbac-proxy" not found
Oct 31 19:16:18.043253 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.043132    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:18.043253 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.043194    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:19.043177362 +0000 UTC m=+40.606647684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:18.143900 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.143708    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:18.144082 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.143903    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 19:16:18.144082 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.143988    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert podName:38ce8722-6108-4502-b83d-0a29a46f88b0 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.143964139 +0000 UTC m=+47.707434475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jcp2j" (UID: "38ce8722-6108-4502-b83d-0a29a46f88b0") : secret "networking-console-plugin-cert" not found
Oct 31 19:16:18.227246 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.227213    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/1.log"
Oct 31 19:16:18.228254 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.228191    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/0.log"
Oct 31 19:16:18.228374 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.228272    2574 generic.go:356] "Generic (PLEG): container finished" podID="a3897285-e4e6-45da-bf32-fcadd1f9b506" containerID="f9ab0191185107cec3cb6699586632b12dd63fd0ad52afbc12b5b9953e787211" exitCode=255
Oct 31 19:16:18.228614 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.228588    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" event={"ID":"a3897285-e4e6-45da-bf32-fcadd1f9b506","Type":"ContainerDied","Data":"f9ab0191185107cec3cb6699586632b12dd63fd0ad52afbc12b5b9953e787211"}
Oct 31 19:16:18.228731 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.228703    2574 scope.go:122] "RemoveContainer" containerID="6acf9bd878fef1281d9d09dab4bf8eb21b2acf31623ace09072c3d39ddf2496e"
Oct 31 19:16:18.228795 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.228756    2574 scope.go:122] "RemoveContainer" containerID="f9ab0191185107cec3cb6699586632b12dd63fd0ad52afbc12b5b9953e787211"
Oct 31 19:16:18.228988 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.228971    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nztjc_openshift-console-operator(a3897285-e4e6-45da-bf32-fcadd1f9b506)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" podUID="a3897285-e4e6-45da-bf32-fcadd1f9b506"
Oct 31 19:16:18.231180 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.231120    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82" event={"ID":"b38ebb5a-bd63-4e8c-94df-c0a904503c4c","Type":"ContainerStarted","Data":"c00202dc0372a188d238d1ad119f6626093b3efd9977af1495afec190592ce69"}
Oct 31 19:16:18.247140 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.247097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:18.247302 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.247200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:18.247302 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.247227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:18.247302 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.247254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:18.247302 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247281    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 19:16:18.247508 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247345    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls podName:b1e328bc-df61-43b5-a5cf-12a2389903a6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.247329879 +0000 UTC m=+47.810800208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9pglj" (UID: "b1e328bc-df61-43b5-a5cf-12a2389903a6") : secret "cluster-monitoring-operator-tls" not found
Oct 31 19:16:18.247508 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.247285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:18.247508 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247370    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:18.247508 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247381    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.247361663 +0000 UTC m=+47.810831987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : configmap references non-existent config key: service-ca.crt
Oct 31 19:16:18.247508 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247392    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f799dcd5-gmv4v: secret "image-registry-tls" not found
Oct 31 19:16:18.247508 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247377    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:18.247508 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247454    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c7865b76-fzmxv: secret "image-registry-tls" not found
Oct 31 19:16:18.247508 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247468    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 19:16:18.247508 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247443    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls podName:74f1221c-9c54-4098-a301-ece68db990eb nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.247427978 +0000 UTC m=+47.810898306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls") pod "image-registry-65f799dcd5-gmv4v" (UID: "74f1221c-9c54-4098-a301-ece68db990eb") : secret "image-registry-tls" not found
Oct 31 19:16:18.247508 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247510    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.247498619 +0000 UTC m=+47.810968935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : secret "router-metrics-certs-default" not found
Oct 31 19:16:18.247937 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247530    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls podName:ae953c47-79ec-4415-9f33-d4fbb364d6c6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.247521608 +0000 UTC m=+47.810991918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls") pod "image-registry-55c7865b76-fzmxv" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6") : secret "image-registry-tls" not found
Oct 31 19:16:18.247937 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.247574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:18.247937 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.247625    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:18.247937 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247643    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 19:16:18.247937 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247696    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls podName:25311b12-0544-48e2-8b00-f56aff90596d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.247682317 +0000 UTC m=+47.811152649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4d9vb" (UID: "25311b12-0544-48e2-8b00-f56aff90596d") : secret "samples-operator-tls" not found
Oct 31 19:16:18.247937 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247753    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 19:16:18.247937 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:18.247820    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls podName:425443c8-69c7-464e-9f85-0578ff4effdc nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.24780348 +0000 UTC m=+47.811273793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls") pod "dns-default-tcmmx" (UID: "425443c8-69c7-464e-9f85-0578ff4effdc") : secret "dns-default-metrics-tls" not found
Oct 31 19:16:18.300533 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.300495    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:18.367687 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.367638    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wl28r-zphd5"]
Oct 31 19:16:18.411001 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:18.410964    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0ae648_a85e_4a86_98f1_db2e9600c67a.slice/crio-531aa693e7a74bc14f4db1b4297bb59e2f1f6e8cee45f437058dd7517a9a06e4 WatchSource:0}: Error finding container 531aa693e7a74bc14f4db1b4297bb59e2f1f6e8cee45f437058dd7517a9a06e4: Status 404 returned error can't find the container with id 531aa693e7a74bc14f4db1b4297bb59e2f1f6e8cee45f437058dd7517a9a06e4
Oct 31 19:16:18.455438 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.455379    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82" podStartSLOduration=406.108334876 podStartE2EDuration="6m52.455361201s" totalImagesPullingTime="6.347026325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:18.455134085 +0000 UTC m=+40.018604435" watchObservedRunningTime="2026-10-31 19:16:18.455361201 +0000 UTC m=+40.018831534"
Oct 31 19:16:18.539191 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.539158    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:18.763412 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:18.763284    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:19.056463 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.056373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:19.056630 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.056544    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:19.056630 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:19.056553    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:19.056771 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:19.056631    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 19:16:19.056771 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:19.056634    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:21.056614263 +0000 UTC m=+42.620084590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:19.056771 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:19.056724    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:21.056699912 +0000 UTC m=+42.620170235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : configmap "kube-rbac-proxy" not found
Oct 31 19:16:19.236106 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.236065    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg" event={"ID":"ac149f85-b47b-4fb7-8ad7-3f985129f292","Type":"ContainerStarted","Data":"e2fd9fa66da679c4009cf8dbda29de8fae07dc63424c97402b034b8c644b5d19"}
Oct 31 19:16:19.237986 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.237950    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wl28r-zphd5" event={"ID":"1b0ae648-a85e-4a86-98f1-db2e9600c67a","Type":"ContainerStarted","Data":"1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3"}
Oct 31 19:16:19.238809 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.238772    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wl28r-zphd5" event={"ID":"1b0ae648-a85e-4a86-98f1-db2e9600c67a","Type":"ContainerStarted","Data":"531aa693e7a74bc14f4db1b4297bb59e2f1f6e8cee45f437058dd7517a9a06e4"}
Oct 31 19:16:19.240036 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.240018    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/1.log"
Oct 31 19:16:19.240466 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.240445    2574 scope.go:122] "RemoveContainer" containerID="f9ab0191185107cec3cb6699586632b12dd63fd0ad52afbc12b5b9953e787211"
Oct 31 19:16:19.240678 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:19.240633    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nztjc_openshift-console-operator(a3897285-e4e6-45da-bf32-fcadd1f9b506)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" podUID="a3897285-e4e6-45da-bf32-fcadd1f9b506"
Oct 31 19:16:19.351463 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.351400    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg" podStartSLOduration=402.203666702 podStartE2EDuration="6m49.351380142s" totalImagesPullingTime="7.14771344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:19.350906884 +0000 UTC m=+40.914377240" watchObservedRunningTime="2026-10-31 19:16:19.351380142 +0000 UTC m=+40.914850473"
Oct 31 19:16:19.453810 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.453749    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hhtc4"]
Oct 31 19:16:19.457046 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.456980    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:19.483965 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.483936    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.484624 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.484598    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v"]
Oct 31 19:16:19.508411 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.508383    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc"]
Oct 31 19:16:19.509845 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.509824    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v"
Oct 31 19:16:19.527280 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.527229    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hhtc4"]
Oct 31 19:16:19.528024 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.527983    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc"
Oct 31 19:16:19.539492 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.539469    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 19:16:19.545452 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.545417    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 19:16:19.545600 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.545473    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5crdh\""
Oct 31 19:16:19.545921 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.545893    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v"]
Oct 31 19:16:19.561362 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.561329    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b92pj\" (UniqueName: \"kubernetes.io/projected/0ec856c2-103c-45fc-af43-bea87251934d-kube-api-access-b92pj\") pod \"migrator-66df5d496b-ldkvc\" (UID: \"0ec856c2-103c-45fc-af43-bea87251934d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc"
Oct 31 19:16:19.561552 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.561372    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xd5w\" (UniqueName: \"kubernetes.io/projected/91815a3a-773d-42b7-8777-d52bb36e0c8e-kube-api-access-8xd5w\") pod \"migrator-66df5d496b-vzd5v\" (UID: \"91815a3a-773d-42b7-8777-d52bb36e0c8e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v"
Oct 31 19:16:19.561552 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.561516    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6vs\" (UniqueName: \"kubernetes.io/projected/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-kube-api-access-6d6vs\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.561674 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.561556    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-key\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.561790 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.561771    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-cabundle\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.585332 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.585300    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc"]
Oct 31 19:16:19.593995 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.593938    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wl28r-zphd5" podStartSLOduration=2.593923883 podStartE2EDuration="2.593923883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:19.593368449 +0000 UTC m=+41.156838785" watchObservedRunningTime="2026-10-31 19:16:19.593923883 +0000 UTC m=+41.157394214"
Oct 31 19:16:19.641252 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.641132    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 19:16:19.641252 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.641152    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 19:16:19.641252 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.641186    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:19.641252 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.641168    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 19:16:19.644747 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.644720    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-htdcp\""
Oct 31 19:16:19.663868 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.662624    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-key\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.665441 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.664531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-cabundle\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.665441 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.664608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b92pj\" (UniqueName: \"kubernetes.io/projected/0ec856c2-103c-45fc-af43-bea87251934d-kube-api-access-b92pj\") pod \"migrator-66df5d496b-ldkvc\" (UID: \"0ec856c2-103c-45fc-af43-bea87251934d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc"
Oct 31 19:16:19.665441 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.664686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xd5w\" (UniqueName: \"kubernetes.io/projected/91815a3a-773d-42b7-8777-d52bb36e0c8e-kube-api-access-8xd5w\") pod \"migrator-66df5d496b-vzd5v\" (UID: \"91815a3a-773d-42b7-8777-d52bb36e0c8e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v"
Oct 31 19:16:19.665441 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.664765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6vs\" (UniqueName: \"kubernetes.io/projected/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-kube-api-access-6d6vs\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.665441 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.665284    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-cabundle\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.669146 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.669118    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-key\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.711402 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.711369    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:19.721815 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.721781    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xd5w\" (UniqueName: \"kubernetes.io/projected/91815a3a-773d-42b7-8777-d52bb36e0c8e-kube-api-access-8xd5w\") pod \"migrator-66df5d496b-vzd5v\" (UID: \"91815a3a-773d-42b7-8777-d52bb36e0c8e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v"
Oct 31 19:16:19.721992 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.721969    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6vs\" (UniqueName: \"kubernetes.io/projected/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-kube-api-access-6d6vs\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.722708 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.722686    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b92pj\" (UniqueName: \"kubernetes.io/projected/0ec856c2-103c-45fc-af43-bea87251934d-kube-api-access-b92pj\") pod \"migrator-66df5d496b-ldkvc\" (UID: \"0ec856c2-103c-45fc-af43-bea87251934d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc"
Oct 31 19:16:19.816222 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.816187    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:19.835195 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.835158    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v"
Oct 31 19:16:19.858026 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.857993    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc"
Oct 31 19:16:19.947369 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:19.947279    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:20.223019 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:20.222987    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fad5f6e_d422_4bc0_87c5_fa1b5be1e2cd.slice/crio-85128312e1bbe8105b928671ac449c454052ebb8e3f60ee061ccb2cd533ba67a WatchSource:0}: Error finding container 85128312e1bbe8105b928671ac449c454052ebb8e3f60ee061ccb2cd533ba67a: Status 404 returned error can't find the container with id 85128312e1bbe8105b928671ac449c454052ebb8e3f60ee061ccb2cd533ba67a
Oct 31 19:16:20.223507 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.223481    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hhtc4"]
Oct 31 19:16:20.251471 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.251206    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5rjx" event={"ID":"c70d21f3-352d-45fc-853c-a74f690d1898","Type":"ContainerStarted","Data":"931ab41f7af0b80e756beab244fc6e534f303a48b9659b62285891875003ecff"}
Oct 31 19:16:20.251471 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.251385    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:20.252717 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.252643    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4" event={"ID":"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd","Type":"ContainerStarted","Data":"85128312e1bbe8105b928671ac449c454052ebb8e3f60ee061ccb2cd533ba67a"}
Oct 31 19:16:20.267208 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.267176    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v"]
Oct 31 19:16:20.273842 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:20.273798    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91815a3a_773d_42b7_8777_d52bb36e0c8e.slice/crio-be1f38ab23685126c139f586888abbfe0c4cfe0789cc69eb6a9103b478468080 WatchSource:0}: Error finding container be1f38ab23685126c139f586888abbfe0c4cfe0789cc69eb6a9103b478468080: Status 404 returned error can't find the container with id be1f38ab23685126c139f586888abbfe0c4cfe0789cc69eb6a9103b478468080
Oct 31 19:16:20.300071 ip-10-0-128-104 kubenswrapper[2574]: W1031 19:16:20.300038    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec856c2_103c_45fc_af43_bea87251934d.slice/crio-f1000233d40b37980090b176413ac3386ee37d5c9fe2d6baf970214632649ef5 WatchSource:0}: Error finding container f1000233d40b37980090b176413ac3386ee37d5c9fe2d6baf970214632649ef5: Status 404 returned error can't find the container with id f1000233d40b37980090b176413ac3386ee37d5c9fe2d6baf970214632649ef5
Oct 31 19:16:20.301038 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.300922    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc"]
Oct 31 19:16:20.367581 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.366776    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g5rjx" podStartSLOduration=35.304287391 podStartE2EDuration="41.366757706s" totalImagesPullingTime="6.062470315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:20.366075102 +0000 UTC m=+41.929545432" watchObservedRunningTime="2026-10-31 19:16:20.366757706 +0000 UTC m=+41.930228040"
Oct 31 19:16:20.516617 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.516522    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:20.724895 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.724866    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:20.725061 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.724923    2574 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:20.725276 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:20.725264    2574 scope.go:122] "RemoveContainer" containerID="f9ab0191185107cec3cb6699586632b12dd63fd0ad52afbc12b5b9953e787211"
Oct 31 19:16:20.725465 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:20.725447    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nztjc_openshift-console-operator(a3897285-e4e6-45da-bf32-fcadd1f9b506)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" podUID="a3897285-e4e6-45da-bf32-fcadd1f9b506"
Oct 31 19:16:21.078204 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:21.077997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:21.078204 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:21.078109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:21.078443 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:21.078313    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:21.078443 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:21.078380    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:25.078360509 +0000 UTC m=+46.641830833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:21.078443 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:21.078432    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 19:16:21.078576 ip-10-0-128-104 kubenswrapper[2574]: E1031 19:16:21.078464    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:25.078453442 +0000 UTC m=+46.641923766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : configmap "kube-rbac-proxy" not found
Oct 31 19:16:21.258134 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:21.258101    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc" event={"ID":"0ec856c2-103c-45fc-af43-bea87251934d","Type":"ContainerStarted","Data":"f1000233d40b37980090b176413ac3386ee37d5c9fe2d6baf970214632649ef5"}
Oct 31 19:16:21.259387 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:21.259355    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v" event={"ID":"91815a3a-773d-42b7-8777-d52bb36e0c8e","Type":"ContainerStarted","Data":"be1f38ab23685126c139f586888abbfe0c4cfe0789cc69eb6a9103b478468080"}
Oct 31 19:16:21.261025 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:21.260997    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4" event={"ID":"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd","Type":"ContainerStarted","Data":"2adf56325f70bd06094357c19358d3c5983d97273a640b3646b54118d37ef4cf"}
Oct 31 19:16:21.262819 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:21.262792    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vs8zn" event={"ID":"0c39943b-0a43-4184-b616-dda8d9f8545e","Type":"ContainerStarted","Data":"cac12a81a974b58fdac41e5037e8ea5b043465ca5cac722e72fd3e65644eecac"}
Oct 31 19:16:21.367221 ip-10-0-128-104 kubenswrapper[2574]: I1031 19:16:21.366952    2574 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:16:21.367102 ip-10-0-128-104 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 19:16:21.386235 ip-10-0-128-104 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 19:16:21.386497 ip-10-0-128-104 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 19:16:21.386569 ip-10-0-128-104 systemd[1]: kubelet.service: Consumed 3.362s CPU time.
Oct 31 19:16:21.396482 ip-10-0-128-104 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 19:16:21.501444 ip-10-0-128-104 kubenswrapper[5006]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:16:21.501444 ip-10-0-128-104 kubenswrapper[5006]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 19:16:21.501444 ip-10-0-128-104 kubenswrapper[5006]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:16:21.501444 ip-10-0-128-104 kubenswrapper[5006]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:16:21.503603 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503516    5006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:16:21.503603 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503602    5006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:16:21.503603 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503607    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503611    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503614    5006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503617    5006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503620    5006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503623    5006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503625    5006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503628    5006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503631    5006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503634    5006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503636    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503639    5006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503642    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503644    5006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503664    5006 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503676    5006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503681    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503685    5006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503688    5006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:16:21.503725 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503691    5006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503694    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503698    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503700    5006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503703    5006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503706    5006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503709    5006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503711    5006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503715    5006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503717    5006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503720    5006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503724    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503736    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503739    5006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503741    5006 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503744    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503750    5006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503753    5006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503757    5006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:16:21.504183 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503762    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503766    5006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503770    5006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503773    5006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503776    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503778    5006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503781    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503784    5006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503786    5006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503789    5006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503793    5006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503797    5006 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503800    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503803    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503806    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503809    5006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503812    5006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503815    5006 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503817    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:16:21.504669 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503821    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503823    5006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503826    5006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503828    5006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503832    5006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503837    5006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503840    5006 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503843    5006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503846    5006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503848    5006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503851    5006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503855    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503858    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503860    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503863    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503866    5006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503868    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503871    5006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503873    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:16:21.505134 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503876    5006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503878    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503881    5006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503883    5006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503886    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503888    5006 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503891    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503894    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503897    5006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503900    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503902    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503905    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503908    5006 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503911    5006 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503913    5006 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503916    5006 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503918    5006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503921    5006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503924    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503926    5006 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:16:21.505604 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503929    5006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503932    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503934    5006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503937    5006 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503940    5006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503943    5006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503946    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.503948    5006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504785    5006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504794    5006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504797    5006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504800    5006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504803    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504806    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504809    5006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504811    5006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504814    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504817    5006 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504819    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504822    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:16:21.506174 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504824    5006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504827    5006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504830    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504833    5006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504836    5006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504838    5006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504841    5006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504843    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504846    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504849    5006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504852    5006 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504854    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504857    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504860    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504863    5006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504865    5006 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504868    5006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504871    5006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504874    5006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:16:21.506698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504877    5006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504879    5006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504882    5006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504884    5006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504887    5006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504889    5006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504892    5006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504895    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504900    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504903    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504905    5006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504908    5006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504911    5006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504913    5006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504934    5006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504938    5006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504941    5006 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504944    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504947    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504950    5006 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:16:21.507181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504953    5006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504956    5006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504959    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504962    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504964    5006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504967    5006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504969    5006 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504972    5006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504975    5006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504978    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504981    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504983    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504990    5006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504993    5006 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504996    5006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.504999    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505002    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505004    5006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505007    5006 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505009    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:16:21.507781 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505013    5006 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505017    5006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505022    5006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505025    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505027    5006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505030    5006 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505033    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505035    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505038    5006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505041    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505044    5006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505046    5006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505049    5006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505051    5006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505054    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505058    5006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505062    5006 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505065    5006 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505067    5006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:16:21.508466 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505070    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505072    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505075    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505078    5006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505080    5006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505083    5006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505087    5006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505090    5006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505093    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505096    5006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505099    5006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505102    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505105    5006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505109    5006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505112    5006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505114    5006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505775    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505786    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505790    5006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:16:21.509038 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505793    5006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505796    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505799    5006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505802    5006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505804    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505807    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505810    5006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505813    5006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505816    5006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505819    5006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505823    5006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505827    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505830    5006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505832    5006 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505835    5006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505837    5006 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505840    5006 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505843    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505846    5006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505850    5006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:16:21.509512 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505853    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505856    5006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505859    5006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505861    5006 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505864    5006 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505867    5006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505870    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505872    5006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505875    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505878    5006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505880    5006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505883    5006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505885    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505888    5006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505891    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505893    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505896    5006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505899    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505901    5006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:16:21.510067 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505905    5006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505910    5006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505913    5006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505916    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505918    5006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505920    5006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505923    5006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505926    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505929    5006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505931    5006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505934    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505936    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505939    5006 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505942    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505944    5006 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505947    5006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505950    5006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505952    5006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505955    5006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505957    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:16:21.510593 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505960    5006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505962    5006 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505965    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505968    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505970    5006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505973    5006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505975    5006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505977    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505980    5006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505982    5006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505985    5006 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505987    5006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505990    5006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505992    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505995    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.505997    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506000    5006 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506002    5006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506005    5006 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506008    5006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:16:21.511530 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506011    5006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506013    5006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506016    5006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506019    5006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506022    5006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506026    5006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506031    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506037    5006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506041    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506045    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506048    5006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506051    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506053    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506056    5006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506059    5006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506061    5006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506064    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506067    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506070    5006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:16:21.512209 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506073    5006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506075    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506078    5006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506080    5006 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506083    5006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506180    5006 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506193    5006 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506200    5006 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506205    5006 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506210    5006 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506213    5006 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506218    5006 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506222    5006 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506226    5006 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506229    5006 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506233    5006 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506236    5006 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506239    5006 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506242    5006 flags.go:64] FLAG: --cgroup-root=""
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506246    5006 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506251    5006 flags.go:64] FLAG: --client-ca-file=""
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506254    5006 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506257    5006 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 19:16:21.512709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506262    5006 flags.go:64] FLAG: --cluster-domain=""
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506265    5006 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506268    5006 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506271    5006 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506275    5006 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506279    5006 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506282    5006 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506285    5006 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506288    5006 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506292    5006 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506295    5006 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506298    5006 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506301    5006 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506304    5006 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506309    5006 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506312    5006 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506315    5006 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506318    5006 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506321    5006 flags.go:64] FLAG: --enable-server="true"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506324    5006 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506328    5006 flags.go:64] FLAG: --event-burst="100"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506331    5006 flags.go:64] FLAG: --event-qps="50"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506334    5006 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506338    5006 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 19:16:21.513258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506341    5006 flags.go:64] FLAG: --eviction-hard=""
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506345    5006 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506348    5006 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506351    5006 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506354    5006 flags.go:64] FLAG: --eviction-soft=""
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506358    5006 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506361    5006 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506364    5006 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506367    5006 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506371    5006 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506374    5006 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506377    5006 flags.go:64] FLAG: --feature-gates=""
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506381    5006 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506384    5006 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506387    5006 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506391    5006 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506394    5006 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506397    5006 flags.go:64] FLAG: --help="false"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506400    5006 flags.go:64] FLAG: --hostname-override="ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506403    5006 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506407    5006 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506410    5006 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506415    5006 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 19:16:21.513892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506420    5006 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506425    5006 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506433    5006 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506437    5006 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506442    5006 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506451    5006 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506454    5006 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506458    5006 flags.go:64] FLAG: --kube-reserved=""
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506461    5006 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506464    5006 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506467    5006 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506470    5006 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506473    5006 flags.go:64] FLAG: --lock-file=""
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506476    5006 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506479    5006 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506482    5006 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506488    5006 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506492    5006 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506495    5006 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506498    5006 flags.go:64] FLAG: --logging-format="text"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506501    5006 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506504    5006 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506507    5006 flags.go:64] FLAG: --manifest-url=""
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506510    5006 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506515    5006 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 19:16:21.514463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506518    5006 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506522    5006 flags.go:64] FLAG: --max-pods="110"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506525    5006 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506528    5006 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506531    5006 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506534    5006 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506537    5006 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506540    5006 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506544    5006 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506553    5006 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506556    5006 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506560    5006 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506563    5006 flags.go:64] FLAG: --pod-cidr=""
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506566    5006 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506569    5006 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506572    5006 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506575    5006 flags.go:64] FLAG: --port="10250"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506579    5006 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506582    5006 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-06ad3bcc727258e01"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506585    5006 flags.go:64] FLAG: --qos-reserved=""
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506588    5006 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506591    5006 flags.go:64] FLAG: --register-node="true"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506594    5006 flags.go:64] FLAG: --register-with-taints=""
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506598    5006 flags.go:64] FLAG: --registry-burst="10"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506601    5006 flags.go:64] FLAG: --registry-qps="5"
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506604    5006 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 19:16:21.515088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506607    5006 flags.go:64] FLAG: --reserved-memory=""
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506611    5006 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506614    5006 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506618    5006 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506621    5006 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506624    5006 flags.go:64] FLAG: --runonce="false"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506627    5006 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506630    5006 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506633    5006 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506636    5006 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506638    5006 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506642    5006 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506645    5006 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506663    5006 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506668    5006 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506674    5006 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506677    5006 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506680    5006 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506683    5006 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506686    5006 flags.go:64] FLAG: --system-cgroups=""
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506691    5006 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506697    5006 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506700    5006 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506703    5006 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506708    5006 flags.go:64] FLAG: --tls-min-version=""
Oct 31 19:16:21.515747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506711    5006 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506714    5006 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506717    5006 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506720    5006 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506723    5006 flags.go:64] FLAG: --v="2"
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506734    5006 flags.go:64] FLAG: --version="false"
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506739    5006 flags.go:64] FLAG: --vmodule=""
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506743    5006 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.506747    5006 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506897    5006 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506901    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506904    5006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506907    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506910    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506912    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506915    5006 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506918    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506921    5006 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506923    5006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506926    5006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506929    5006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506931    5006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:16:21.516384 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506935    5006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506938    5006 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506941    5006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506943    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506946    5006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506950    5006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506953    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506956    5006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506958    5006 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506961    5006 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506964    5006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506966    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506969    5006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506971    5006 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506974    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506977    5006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506979    5006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506982    5006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506985    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506988    5006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:16:21.517007 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506990    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506993    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506995    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.506998    5006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507001    5006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507003    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507006    5006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507008    5006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507011    5006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507013    5006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507016    5006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507018    5006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507022    5006 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507025    5006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507028    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507031    5006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507033    5006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507038    5006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507040    5006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507043    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:16:21.517487 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507046    5006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507048    5006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507051    5006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507053    5006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507056    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507059    5006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507062    5006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507065    5006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507068    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507071    5006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507074    5006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507077    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507079    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507081    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507084    5006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507086    5006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507089    5006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507091    5006 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507094    5006 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:16:21.518056 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507096    5006 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507099    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507102    5006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507104    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507107    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507110    5006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507114    5006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507118    5006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507121    5006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507124    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507128    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507130    5006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507134    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507136    5006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507139    5006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507143    5006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507146    5006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507150    5006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507153    5006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:16:21.518896 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507156    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507158    5006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507161    5006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507164    5006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507166    5006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507170    5006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507172    5006 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507175    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507177    5006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507180    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507183    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507185    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507188    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507191    5006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507194    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.507219    5006 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:16:21.519405 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507259    5006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507263    5006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507265    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507269    5006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507272    5006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507274    5006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507277    5006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507281    5006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507284    5006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507287    5006 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507289    5006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507292    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507294    5006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507297    5006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507300    5006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507302    5006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507305    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507307    5006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507311    5006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:16:21.519843 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507314    5006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507316    5006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507319    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507322    5006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507325    5006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507327    5006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507330    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507333    5006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507335    5006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507338    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507340    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507343    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507345    5006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507348    5006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507350    5006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507353    5006 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507355    5006 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507358    5006 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507360    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507363    5006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:16:21.520423 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507367    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507370    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507372    5006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507377    5006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507381    5006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507384    5006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507387    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507391    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507394    5006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507397    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507399    5006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507402    5006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507406    5006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507409    5006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507412    5006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507414    5006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507418    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507421    5006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507423    5006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:16:21.521119 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507426    5006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507428    5006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507431    5006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507434    5006 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507436    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507439    5006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507441    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507444    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507446    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507449    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507451    5006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507454    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507456    5006 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507460    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507462    5006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507465    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507467    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507470    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507472    5006 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507476    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:16:21.521696 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507478    5006 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507481    5006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507483    5006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507486    5006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507488    5006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507491    5006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507497    5006 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507501    5006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507505    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507509    5006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507514    5006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507517    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507521    5006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507526    5006 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507529    5006 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507533    5006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507537    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507542    5006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507545    5006 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507547    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:16:21.522436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507550    5006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507553    5006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507555    5006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507558    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507560    5006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507564    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507567    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.507570    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.512642    5006 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.512684    5006 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.512704    5006 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.512709    5006 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512792    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512798    5006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512802    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512805    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512808    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512812    5006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512815    5006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:16:21.522992 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512818    5006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512821    5006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512823    5006 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512826    5006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512829    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512832    5006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512836    5006 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512838    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512841    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512844    5006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512847    5006 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512849    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512852    5006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512855    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512857    5006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512860    5006 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512863    5006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512865    5006 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512868    5006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512871    5006 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:16:21.523459 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512874    5006 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512877    5006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512880    5006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512882    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512885    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512888    5006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512890    5006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512893    5006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512896    5006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512898    5006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512901    5006 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512904    5006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512907    5006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512909    5006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512912    5006 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512914    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512917    5006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512920    5006 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512923    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512926    5006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:16:21.523993 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512929    5006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512933    5006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512937    5006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512940    5006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512942    5006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512945    5006 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512947    5006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512950    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512953    5006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512956    5006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512958    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512962    5006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512967    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512970    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512973    5006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512975    5006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512978    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512980    5006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512983    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:16:21.524470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512986    5006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512989    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512992    5006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512994    5006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.512997    5006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513000    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513003    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513005    5006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513008    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513011    5006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513014    5006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513016    5006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513019    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513022    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513025    5006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513027    5006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513030    5006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513032    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513034    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:16:21.524953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513037    5006 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513040    5006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513042    5006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513045    5006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513047    5006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513050    5006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513052    5006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513055    5006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513057    5006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513060    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513062    5006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513065    5006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513068    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513070    5006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513073    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513076    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513079    5006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513081    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513084    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513086    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:16:21.525410 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513089    5006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.513118    5006 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513270    5006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513275    5006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513278    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513281    5006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513284    5006 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513288    5006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513290    5006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513293    5006 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513296    5006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513298    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513301    5006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513304    5006 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513306    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513309    5006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:16:21.526024 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513311    5006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513314    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513317    5006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513319    5006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513322    5006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513324    5006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513327    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513330    5006 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513332    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513335    5006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513337    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513340    5006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513343    5006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513345    5006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513348    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513351    5006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513353    5006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513357    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513359    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513362    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:16:21.526720 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513364    5006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513366    5006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513369    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513372    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513375    5006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513377    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513380    5006 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513382    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513385    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513390    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513393    5006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513395    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513398    5006 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513401    5006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513404    5006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513407    5006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513409    5006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513412    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513414    5006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513417    5006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:16:21.527486 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513420    5006 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513422    5006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513425    5006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513428    5006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513430    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513433    5006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513436    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513439    5006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513444    5006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513447    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513449    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513452    5006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513454    5006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513457    5006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513460    5006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513462    5006 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513465    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513468    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513470    5006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:16:21.528075 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513473    5006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513475    5006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513479    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513482    5006 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513485    5006 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513488    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513490    5006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513493    5006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513495    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513509    5006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513512    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513515    5006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513518    5006 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513520    5006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513523    5006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513525    5006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513528    5006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513531    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513535    5006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:16:21.528541 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513539    5006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513542    5006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513545    5006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513548    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513551    5006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513553    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513556    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513559    5006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513561    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513564    5006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513567    5006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513569    5006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513573    5006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513575    5006 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.513599    5006 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:16:21.529106 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513632    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513641    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513644    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513662    5006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513667    5006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513671    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513674    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513677    5006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513679    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513682    5006 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513685    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513687    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513690    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513693    5006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513695    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513698    5006 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513700    5006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513704    5006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513706    5006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513709    5006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:16:21.529478 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513711    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513714    5006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513716    5006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513719    5006 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513722    5006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513724    5006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513727    5006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513729    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513732    5006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513734    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513739    5006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513744    5006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513747    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513751    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513755    5006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513757    5006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513760    5006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513763    5006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513765    5006 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:16:21.530044 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513768    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513771    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513773    5006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513776    5006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513779    5006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513781    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513784    5006 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513786    5006 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513789    5006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513791    5006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513794    5006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513796    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513799    5006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513802    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513804    5006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513807    5006 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513810    5006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513812    5006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513815    5006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513817    5006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:16:21.530498 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513820    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513824    5006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513827    5006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513830    5006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513834    5006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513837    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513841    5006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513843    5006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513846    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513848    5006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513851    5006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513854    5006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513857    5006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513860    5006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513862    5006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513865    5006 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513868    5006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513870    5006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513873    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:16:21.531004 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513876    5006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513878    5006 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513881    5006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513883    5006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513886    5006 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513888    5006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513891    5006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513894    5006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513898    5006 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513901    5006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513904    5006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513906    5006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513909    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513912    5006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513914    5006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513917    5006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513919    5006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513922    5006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513926    5006 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513930    5006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:16:21.531467 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513932    5006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513935    5006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513937    5006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513940    5006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513942    5006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513946    5006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513949    5006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:21.513952    5006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.514027    5006 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.515065    5006 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.515124    5006 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.515312    5006 server.go:1014] "Starting client certificate rotation"
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.515443    5006 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.515502    5006 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:10:38 +0000 UTC" deadline="2028-03-27 05:28:43.399005274 +0000 UTC"
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.515530    5006 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12298h12m21.883477099s"
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.515813    5006 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:16:21.532002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.516961    5006 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:16:21.532434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.518391    5006 log.go:25] "Validated CRI v1 runtime API"
Oct 31 19:16:21.532434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.520069    5006 log.go:25] "Validated CRI v1 image API"
Oct 31 19:16:21.532434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.520860    5006 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 19:16:21.532434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.522865    5006 fs.go:135] Filesystem UUIDs: map[0831209d-8d10-45b6-800e-1fa7fb140c46:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 df679879-3c18-4193-ac99-c272e05e4fc3:/dev/nvme0n1p3]
Oct 31 19:16:21.532574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.522887    5006 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3809a62c76782d05a5ab1b565d2b39220002200918ebeebac00247553e17b9f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3809a62c76782d05a5ab1b565d2b39220002200918ebeebac00247553e17b9f8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4823da1de91f8837656168f3d4b3a9fb932075e173848fef95de8ac51c0a5d99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4823da1de91f8837656168f3d4b3a9fb932075e173848fef95de8ac51c0a5d99/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/531aa693e7a74bc14f4db1b4297bb59e2f1f6e8cee45f437058dd7517a9a06e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/531aa693e7a74bc14f4db1b4297bb59e2f1f6e8cee45f437058dd7517a9a06e4/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/540a2d1bf2708ed1dd233abc7be4a810bb08e76c08a0d59cecb238d10cb3a063/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/540a2d1bf2708ed1dd233abc7be4a810bb08e76c08a0d59cecb238d10cb3a063/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b253487198586d7fe47bf2bad1a818744ac7de28b2a03b8843a2637d06345e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b253487198586d7fe47bf2bad1a818744ac7de28b2a03b8843a2637d06345e4/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82dcbd39451e591ca43cf604e1f62e55fef66d5f326889e7867d1447a7ee2a93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82dcbd39451e591ca43cf604e1f62e55fef66d5f326889e7867d1447a7ee2a93/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85128312e1bbe8105b928671ac449c454052ebb8e3f60ee061ccb2cd533ba67a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85128312e1bbe8105b928671ac449c454052ebb8e3f60ee061ccb2cd533ba67a/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cb1a8dd7005964caeb03488c7f4b3620912c1271d6c2db36698fa0c378bd691/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cb1a8dd7005964caeb03488c7f4b3620912c1271d6c2db36698fa0c378bd691/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d38d9fda59af58fef3810194e579b4f6d68ea4c7405dfc86ba96eb8d913fb00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d38d9fda59af58fef3810194e579b4f6d68ea4c7405dfc86ba96eb8d913fb00/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2e6eed54b1b98f5f4f90c251ba8ed62f3164a28611fcd0f351f42159ecf8c38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2e6eed54b1b98f5f4f90c251ba8ed62f3164a28611fcd0f351f42159ecf8c38/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4040f38b2d991ed01cde381142f55f8cb1886d0d151d3b704c0aad4cf754caf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4040f38b2d991ed01cde381142f55f8cb1886d0d151d3b704c0aad4cf754caf/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8947f9b17cb096a6d5fb051ce149de3bbd1a6efa6d854d62591418947562cc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8947f9b17cb096a6d5fb051ce149de3bbd1a6efa6d854d62591418947562cc9/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac545f00e57d82c65d2fbd08eb776c5127df5595567899a4dd1c433595badb75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac545f00e57d82c65d2fbd08eb776c5127df5595567899a4dd1c433595badb75/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5739c8cbd0f5e746ddb218c574abe4d33dc870cbb2084e4effce2ddb298fc53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5739c8cbd0f5e746ddb218c574abe4d33dc870cbb2084e4effce2ddb298fc53/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbc6dc8dea02d76785975ab983ade811897aac5fc2b9b2a2727b82e28ec2f61f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbc6dc8dea02d76785975ab983ade811897aac5fc2b9b2a2727b82e28ec2f61f/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be1f38ab23685126c139f586888abbfe0c4cfe0789cc69eb6a9103b478468080/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be1f38ab23685126c139f586888abbfe0c4cfe0789cc69eb6a9103b478468080/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca557849d78f9ca1965ca46ec0e7e6d7a7174dcdd63d56f448038f9d1e1951e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca557849d78f9ca1965ca46ec0e7e6d7a7174dcdd63d56f448038f9d1e1951e8/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbf00387d7f8b5f214ba89e3e762a65b0ad63bb748bd0d21c0b56eb9d6da04ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbf00387d7f8b5f214ba89e3e762a65b0ad63bb748bd0d21c0b56eb9d6da04ea/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce53bbfb2bf87bb9146509806c20754fffd7ff4bf7ca58db50ee758ac749a479/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce53bbfb2bf87bb9146509806c20754fffd7ff4bf7ca58db50ee758ac749a479/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d07a0e8ed908aae0ee04fa0954af3ec07c9f6d0f461e0ee4376735304279ee35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d07a0e8ed908aae0ee04fa0954af3ec07c9f6d0f461e0ee4376735304279ee35/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dabb61cab0ada53f5bdc5279952acc1249bacd3595d93d0fa33e7ba42dfa85d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dabb61cab0ada53f5bdc5279952acc1249bacd3595d93d0fa33e7ba42dfa85d7/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1000233d40b37980090b176413ac3386ee37d5c9fe2d6baf970214632649ef5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1000233d40b37980090b176413ac3386ee37d5c9fe2d6baf970214632649ef5/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2e931f8dde77690c89d31097dc30f8fc188ce421aa57ca6e2d03bf3e6611313/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2e931f8dde77690c89d31097dc30f8fc188ce421aa57ca6e2d03bf3e6611313/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7ec2ee3429cb367b4b4cfef810d172f37250b5cdd37c27e0a09d54ad6dd104e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7ec2ee3429cb367b4b4cfef810d172f37250b5cdd37c27e0a09d54ad6dd104e/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/facc7d3c43fa65af9418b01f85949f531d104e9625d6ddc915228cbcf4362306/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/facc7d3c43fa65af9418b01f85949f531d104e9625d6ddc915228cbcf4362306/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc1ea5cc45e4f1d27a8ea54a3a33042f9120ccb107860f4dbc52028e9fe25f39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc1ea5cc45e4f1d27a8ea54a3a33042f9120ccb107860f4dbc52028e9fe25f39/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c39943b-0a43-4184-b616-dda8d9f8545e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0c39943b-0a43-4184-b616-dda8d9f8545e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ec856c2-103c-45fc-af43-bea87251934d/volumes/kubernetes.io~projected/kube-api-access-b92pj:{mountpoint:/var/lib/kubelet/pods/0ec856c2-103c-45fc-af43-bea87251934d/volumes/kubernetes.io~projected/kube-api-access-b92pj major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd/volumes/kubernetes.io~projected/kube-api-access-6d6vs:{mountpoint:/var/lib/kubelet/pods/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd/volumes/kubernetes.io~projected/kube-api-access-6d6vs major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd/volumes/kubernetes.io~secret/signing-key major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1297e6ef-5035-497c-9777-c6b4448bceba/volumes/kubernetes.io~projected/kube-api-access-f6x56:{mountpoint:/var/lib/kubelet/pods/1297e6ef-5035-497c-9777-c6b4448bceba/volumes/kubernetes.io~projected/kube-api-access-f6x56 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b0ae648-a85e-4a86-98f1-db2e9600c67a/volumes/kubernetes.io~projected/kube-api-access-knvdl:{mountpoint:/var/lib/kubelet/pods/1b0ae648-a85e-4a86-98f1-db2e9600c67a/volumes/kubernetes.io~projected/kube-api-access-knvdl major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c72eeb1-17c0-4dd4-99c3-2509ba36b668/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1c72eeb1-17c0-4dd4-99c3-2509ba36b668/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c72eeb1-17c0-4dd4-99c3-2509ba36b668/volumes/kubernetes.io~projected/kube-api-access-xp6kc:{mountpoint:/var/lib/kubelet/pods/1c72eeb1-17c0-4dd4-99c3-2509ba36b668/volumes/kubernetes.io~projected/kube-api-access-xp6kc major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c72eeb1-17c0-4dd4-99c3-2509ba36b668/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1c72eeb1-17c0-4dd4-99c3-2509ba36b668/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f59bf5f-da15-4174-a5be-b843478f0191/volumes/kubernetes.io~projected/kube-api-access-792p5:{mountpoint:/var/lib/kubelet/pods/1f59bf5f-da15-4174-a5be-b843478f0191/volumes/kubernetes.io~projected/kube-api-access-792p5 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f59bf5f-da15-4174-a5be-b843478f0191/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1f59bf5f-da15-4174-a5be-b843478f0191/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25311b12-0544-48e2-8b00-f56aff90596d/volumes/kubernetes.io~projected/kube-api-access-sfd9g:{mountpoint:/var/lib/kubelet/pods/25311b12-0544-48e2-8b00-f56aff90596d/volumes/kubernetes.io~projected/kube-api-access-sfd9g major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/291cf47e-f759-4489-abb5-1bafffff7b25/volumes/kubernetes.io~projected/kube-api-access-nqdqq:{mountpoint:/var/lib/kubelet/pods/291cf47e-f759-4489-abb5-1bafffff7b25/volumes/kubernetes.io~projected/kube-api-access-nqdqq major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/425443c8-69c7-464e-9f85-0578ff4effdc/volumes/kubernetes.io~projected/kube-api-access-w7f9m:{mountpoint:/var/lib/kubelet/pods/425443c8-69c7-464e-9f85-0578ff4effdc/volumes/kubernetes.io~projected/kube-api-access-w7f9m major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7/volumes/kubernetes.io~projected/kube-api-access-s5q49:{mountpoint:/var/lib/kubelet/pods/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7/volumes/kubernetes.io~projected/kube-api-access-s5q49 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e36ee6a-f51a-4182-8bfb-68da77946954/volumes/kubernetes.io~projected/kube-api-access-wcwv2:{mountpoint:/var/lib/kubelet/pods/4e36ee6a-f51a-4182-8bfb-68da77946954/volumes/kubernetes.io~projected/kube-api-access-wcwv2 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50011131-90de-4df2-ba1e-28b2a919dbf0/volumes/kubernetes.io~projected/kube-api-access-gtnkn:{mountpoint:/var/lib/kubelet/pods/50011131-90de-4df2-ba1e-28b2a919dbf0/volumes/kubernetes.io~projected/kube-api-access-gtnkn major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50011131-90de-4df2-ba1e-28b2a919dbf0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/50011131-90de-4df2-ba1e-28b2a919dbf0/volumes/kubernetes.io~secret/cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/613da58f-accf-4a50-a1b3-b7dccc52fae7/volumes/kubernetes.io~projected/kube-api-access-km84r:{mountpoint:/var/lib/kubelet/pods/613da58f-accf-4a50-a1b3-b7dccc52fae7/volumes/kubernetes.io~projected/kube-api-access-km84r major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73377bc4-2624-4ed6-a5aa-30228b89cbfb/volumes/kubernetes.io~projected/kube-api-access-ls8vr:{mountpoint:/var/lib/kubelet/pods/73377bc4-2624-4ed6-a5aa-30228b89cbfb/volumes/kubernetes.io~projected/kube-api-access-ls8vr major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~projected/kube-api-access-4b9kt:{mountpoint:/var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~projected/kube-api-access-4b9kt major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78ed8e27-2cd2-43e5-8ee1-23042619743e/volumes/kubernetes.io~projected/kube-api-access-v9dh8:{mountpoint:/var/lib/kubelet/pods/78ed8e27-2cd2-43e5-8ee1-23042619743e/volumes/kubernetes.io~projected/kube-api-access-v9dh8 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91815a3a-773d-42b7-8777-d52bb36e0c8e/volumes/kubernetes.io~projected/kube-api-access-8xd5w:{mountpoint:/var/lib/kubelet/pods/91815a3a-773d-42b7-8777-d52bb36e0c8e/volumes/kubernetes.io~projected/kube-api-access-8xd5w major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fe381a8-6a81-4e21-96c6-715ecd22ff45/volumes/kubernetes.io~projected/kube-api-access-fdj6v:{mountpoint:/var/lib/kubelet/pods/9fe381a8-6a81-4e21-96c6-715ecd22ff45/volumes/kubernetes.io~projected/kube-api-access-fdj6v major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4/volumes/kubernetes.io~projected/kube-api-access-g7wg7:{mountpoint:/var/lib/kubelet/pods/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4/volumes/kubernetes.io~projected/kube-api-access-g7wg7 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3897285-e4e6-45da-bf32-fcadd1f9b506/volumes/kubernetes.io~projected/kube-api-access-qjmw8:{mountpoint:/var/lib/kubelet/pods/a3897285-e4e6-45da-bf32-fcadd1f9b506/volumes/kubernetes.io~projected/kube-api-access-qjmw8 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3897285-e4e6-45da-bf32-fcadd1f9b506/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a3897285-e4e6-45da-bf32-fcadd1f9b506/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac149f85-b47b-4fb7-8ad7-3f985129f292/volumes/kubernetes.io~projected/kube-api-access-7k8p8:{mountpoint:/var/lib/kubelet/pods/ac149f85-b47b-4fb7-8ad7-3f985129f292/volumes/kubernetes.io~projected/kube-api-access-7k8p8 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~projected/kube-api-access-fdfnn:{mountpoint:/var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~projected/kube-api-access-fdfnn major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b05731a8-35c3-4afa-97fa-6d3db50533b3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b05731a8-35c3-4afa-97fa-6d3db50533b3/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1e328bc-df61-43b5-a5cf-12a2389903a6/volumes/kubernetes.io~projected/kube-api-access-wwlfc:{mountpoint:/var/lib/kubelet/pods/b1e328bc-df61-43b5-a5cf-12a2389903a6/volumes/kubernetes.io~projected/kube-api-access-wwlfc major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b37ff40c-3db6-4467-aeaa-c23407b5fce5/volumes/kubernetes.io~projected/kube-api-access-4tlwl:{mountpoint:/var/lib/kubelet/pods/b37ff40c-3db6-4467-aeaa-c23407b5fce5/volumes/kubernetes.io~projected/kube-api-access-4tlwl major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b38ebb5a-bd63-4e8c-94df-c0a904503c4c/volumes/kubernetes.io~projected/kube-api-access-pqzf2:{mountpoint:/var/lib/kubelet/pods/b38ebb5a-bd63-4e8c-94df-c0a904503c4c/volumes/kubernetes.io~projected/kube-api-access-pqzf2 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b38ebb5a-bd63-4e8c-94df-c0a904503c4c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b38ebb5a-bd63-4e8c-94df-c0a904503c4c/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0eaee38-fd4e-489a-a854-39245cd595dc/volumes/kubernetes.io~projected/kube-api-access-p8qfj:{mountpoint:/var/lib/kubelet/pods/c0eaee38-fd4e-489a-a854-39245cd595dc/volumes/kubernetes.io~projected/kube-api-access-p8qfj major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1b27776-d610-4efa-9f8e-e08f0161a07f/volumes/kubernetes.io~projected/kube-api-access-prj9t:{mountpoint:/var/lib/kubelet/pods/c1b27776-d610-4efa-9f8e-e08f0161a07f/volumes/kubernetes.io~projected/kube-api-access-prj9t major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1b27776-d610-4efa-9f8e-e08f0161a07f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c1b27776-d610-4efa-9f8e-e08f0161a07f/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c70d21f3-352d-45fc-853c-a74f690d1898/volumes/kubernetes.io~projected/kube-api-access-5wfph:{mountpoint:/var/lib/kubelet/pods/c70d21f3-352d-45fc-853c-a74f690d1898/volumes/kubernetes.io~projected/kube-api-access-5wfph major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40/volumes/kubernetes.io~projected/kube-api-access-njvjg:{mountpoint:/var/lib/kubelet/pods/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40/volumes/kubernetes.io~projected/kube-api-access-njvjg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7/volumes/kubernetes.io~projected/kube-api-access-8v7sm:{mountpoint:/var/lib/kubelet/pods/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7/volumes/kubernetes.io~projected/kube-api-access-8v7sm major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9c1e13d-3f76-4884-af1a-956dc34fd1d6/volumes/kubernetes.io~projected/kube-api-access-v79fc:{mountpoint:/var/lib/kubelet/pods/f9c1e13d-3f76-4884-af1a-956dc34fd1d6/volumes/kubernetes.io~projected/kube-api-access-v79fc major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9c1e13d-3f76-4884-af1a-956dc34fd1d6/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/f9c1e13d-3f76-4884-af1a-956dc34fd1d6/volumes/kubernetes.io~secret/default-certificate major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9c1e13d-3f76-4884-af1a-956dc34fd1d6/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/f9c1e13d-3f76-4884-af1a-956dc34fd1d6/volumes/kubernetes.io~secret/stats-auth major:0 minor:220 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7af8e7e0b55dbbe5531e95355e8f10648f9fa6b107b099a82eb8393ee793e54b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2d40b8cb6b3aaee3a610848c2de9fba7ea0487b6eaf49d05df72d8e48d731251/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e308c4967b4f46b0b69c10ab4fe66059ce0af6a33e9f8965f0ff1648fb2a3849/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7b047bcb0c4576195908f15f1983af2fc76ccd03421b79efb12bcba722fa64aa/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3621eaa4402e1fecd50afac3686cbb36b93fba9e0c2c7292655313506568c5f2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d13129cc75f08a3861f04df6a73efa7bbd9e78c76d790bee623ec693a831b1e5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/01b65f2f7ea8c15c5cd7f4c6f8afe61b577a67dc0e3b7665ff88d68803f5a574/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5f816ac1c27a88a2adc807003146169f944cde3c92a9db58a3ecc82d73e63e26/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/cf9c34fca30e5dc8abd850c84cbfc438c2821adee6d9e9ab8ca02e9edb9c6922/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/fa9252eb5ab4cfad0d72ff474978341bb0323a2a9b083d5b7ca83e20ec1f84fc/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/ebdd0c58e13a219de29f35f81a00a28c489d0829d59028c43df1aa6b81defa7a/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/3b1ff442deecb3252a6efa60b8b565c79226f74f86048ed3248c26c4fa941aea/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/97c269f0deebb2ee781682e96d8c3d84918f7b5bb1b9f5f9560ae8dbcc915983/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/825c4d673b0ce914096927ada3bc3c71510f352a52eb80aae0d29fb3f0284999/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3ae02547a036a76989866ad2ba8781489054aa8794ec048186ef9b177864fa49/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b5bc6680ef2f88c20530e506da9c132c63d382785f83102ce24bcfbe4333582d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/398456508a9ecef3d6bd0b7e218977a3dd01a1984f8e75028d2ff2647a4749f9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/23d715051bb65ef2b43d1759048cd7b90900bcaca6c7b3948ae919604e6ff71f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1f5a921ea677899247c504aaf0ce5e2767eb4fadcec58b53ec375420199094b8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/988b147b0226ce579d76aa5dcd9750e730a5ca17c1e2b49a03ce85698d19ff1d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/de42871c8ed313c39c025cc90ab2913c3d3175d3d870569aabbe38407d5d8a74/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5479d95ff5f1fc9cf212c6ac21616f180eadb08cc8ac541ec6015568892fb788/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/d402a1a93516b158d218c884f3b89ba7c35ffcbd6d1ba2ef4b04bf54145b3b14/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/038c569fe19cd249fe6443b4dd663dfc217d7c30eb68f481a019d04eba7a2bfe/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f6460276c105125c6514de283f35b02a9c2a7fe704d06c6bd2ad8bc6b01dcd28/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9eb852d99389eec638ce3161c7a1d7833a977a4fabc4dbec95e6b0a9ab76999a/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f8d8fe5367b4eb791223bfa86d5a8ec200e193bfd7693abd529318740577892f/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/c2b4788554e2666249dc6aed2b9e9a5c935bed479638f722ad9bd1cd0d92b692/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/e563cb21520cb3739df11f5aa9f1625edfff5af1d34caf8fdd197feb346e2a53/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/c4dfada2f5dd4a0eb85ffe3191ef5746392370998e579f5de5c036e1bdd13e6e/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/d82292f209ead1c53682899cae4345ebde0a23942faa5cc4bfc6347739726114/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/82688b0b731e0df26091348a27ce415d16140323f4895bc0e69f6cd165bf707e/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/6111c5b28f977db0c1271c0e3ad62f269f675852a548b3e2c2d5a20882ec41c9/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/fca5ea8c71a42f2cd0cc032011806dbeb3115fac5290cf66aacd5f89808c3b90/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/129a562da4e262cf2ea35c00ac3be5f71507636e1de7ca8243f3be2b3b208908/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/2492ced7a36c76d452f3e05d68ad55d9d29e5e482e968db2d7cbf0070772f87b/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/37218a8b6d73683375996602334ef22402c909e6a557dec6d51a3f15e811ab72/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/a8ec80a59e4387d1bbff400f70b8a4f91288b64493dc9582bcd41f6364753e31/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/36b3c981df5563dd4661a2eabd124a645a4be17589ebfc4a0dd3d64ddde351c6/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/618094a881ba2a2aac85602d9a905402e9abf87474eea11b0502de542ca560c1/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/851d47cf8ebea740101857af2e8a2096b796d7ee9b053f374fa54899fe9e19a9/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/bc980262d91199e0a30f668b1b8113b315e0f6c01ececab3a0983318e219e403/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/9a26c37adff2404de39cee914999f7a3f33307f6458548c314382cbdf2fc1ebd/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/4b0572fc99775b5f76122acfae1abda34093d0bd7e0b792b436255047224b244/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/707f9c33ece3ebce8503f318088e56daf3ef7c536e912d26e563cae78ff8e081/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/5d16623451853bf1f76f8fb6dbdcf4b16aa77b830e862c36dec6dc6a79864678/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/51629b4cee7e0b496c286871022eeb8248fd6441618bb91969f7eacc5127e5f8/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/d97748cef7dd4b63816b467fe620cf1962ffc1a9739646cf24aac739af431895/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/ea65c84604734c0500b48d5aac436fe8222e163fa7c9366af191f3c9c8434fcb/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/081fc4bc4891b44ac612751f74538167eb7a4f03fe84d15e64f5a0cc78e76221/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/40602bf26128d6756dee169f8a1719bcdbb6d9e01595f02ea52763fa79724b0d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f288484547a66492eb033198e0b4c632a40e8d2babae2505c09cb2fc9d55ec5f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a08a8f3dd0c75cc50b0bae0903c43e1d994785015d261ed21564554a6f17281f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/545488d83aca8be6af76f0bd6961c488f892639330d81989f7e510b95c9e6393/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6942638c4ea19145665a01d07248e9b7426ec58c721f009762054154ec9f1cf1/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/849ddeb2a4021da5af39b0a8b63a869364f73dd0ab99506feab6ac2ab3c8f04d/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2d3c6538f8fdaca1bac8d683b0e50daaa1f073a71c62e138131696aaa9c98958/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/2d8331ff73bf59a278d74b199c25e33192ffb1d25086835b866702a4cbbd62fd/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7bbcea5bc358b36d5f5114b313614f6f81cd04ec0a81ec970e727c68e9cb44cb/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.531679    5006 manager.go:217] Machine: {Timestamp:2026-10-31 19:16:21.531185634 +0000 UTC m=+0.060719678 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099620 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21b719ee2d2e25a4c300e5365301db SystemUUID:ec21b719-ee2d-2e25-a4c3-00e5365301db BootID:29e21b7b-98ac-4ea5-a937-deebc7e2c54e Filesystems:[{Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85128312e1bbe8105b928671ac449c454052ebb8e3f60ee061ccb2cd533ba67a/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4823da1de91f8837656168f3d4b3a9fb932075e173848fef95de8ac51c0a5d99/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1b27776-d610-4efa-9f8e-e08f0161a07f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc1ea5cc45e4f1d27a8ea54a3a33042f9120ccb107860f4dbc52028e9fe25f39/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbf00387d7f8b5f214ba89e3e762a65b0ad63bb748bd0d21c0b56eb9d6da04ea/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73377bc4-2624-4ed6-a5aa-30228b89cbfb/volumes/kubernetes.io~projected/kube-api-access-ls8vr DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4040f38b2d991ed01cde381142f55f8cb1886d0d151d3b704c0aad4cf754caf/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2e931f8dde77690c89d31097dc30f8fc188ce421aa57ca6e2d03bf3e6611313/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0eaee38-fd4e-489a-a854-39245cd595dc/volumes/kubernetes.io~projected/kube-api-access-p8qfj DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7ec2ee3429cb367b4b4cfef810d172f37250b5cdd37c27e0a09d54ad6dd104e/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd/volumes/kubernetes.io~projected/kube-api-access-6d6vs DeviceMajor:0 DeviceMinor:346 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40/volumes/kubernetes.io~projected/kube-api-access-njvjg DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25311b12-0544-48e2-8b00-f56aff90596d/volumes/kubernetes.io~projected/kube-api-access-sfd9g DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be1f38ab23685126c139f586888abbfe0c4cfe0789cc69eb6a9103b478468080/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/291cf47e-f759-4489-abb5-1bafffff7b25/volumes/kubernetes.io~projected/kube-api-access-nqdqq DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/a3897285-e4e6-45da-bf32-fcadd1f9b506/volumes/kubernetes.io~projected/kube-api-access-qjmw8 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9c1e13d-3f76-4884-af1a-956dc34fd1d6/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b37ff40c-3db6-4467-aeaa-c23407b5fce5/volumes/kubernetes.io~projected/kube-api-access-4tlwl DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2e6eed54b1b98f5f4f90c251ba8ed62f3164a28611fcd0f351f42159ecf8c38/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8947f9b17cb096a6d5fb051ce149de3bbd1a6efa6d854d62591418947562cc9/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/425443c8-69c7-464e-9f85-0578ff4effdc/volumes/kubernetes.io~projected/kube-api-access-w7f9m DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82dcbd39451e591ca43cf604e1f62e55fef66d5f326889e7867d1447a7ee2a93/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cb1a8dd7005964caeb03488c7f4b3620912c1271d6c2db36698fa0c378bd691/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dabb61cab0ada53f5bdc5279952acc1249bacd3595d93d0fa33e7ba42dfa85d7/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5739c8cbd0f5e746ddb218c574abe4d33dc870cbb2084e4effce2ddb298fc53/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78ed8e27-2cd2-43e5-8ee1-23042619743e/volumes/kubernetes.io~projected/kube-api-access-v9dh8 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c70d21f3-352d-45fc-853c-a74f690d1898/volumes/kubernetes.io~projected/kube-api-access-5wfph DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c39943b-0a43-4184-b616-dda8d9f8545e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b05731a8-35c3-4afa-97fa-6d3db50533b3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e36ee6a-f51a-4182-8bfb-68da77946954/volumes/kubernetes.io~projected/kube-api-access-wcwv2 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1000233d40b37980090b176413ac3386ee37d5c9fe2d6baf970214632649ef5/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3897285-e4e6-45da-bf32-fcadd1f9b506/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/540a2d1bf2708ed1dd233abc7be4a810bb08e76c08a0d59cecb238d10cb3a063/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac545f00e57d82c65d2fbd08eb776c5127df5595567899a4dd1c433595badb75/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4/volumes/kubernetes.io~projected/kube-api-access-g7wg7 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f59bf5f-da15-4174-a5be-b843478f0191/volumes/kubernetes.io~projected/kube-api-access-792p5 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbc6dc8dea02d76785975ab983ade811897aac5fc2b9b2a2727b82e28ec2f61f/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/613da58f-accf-4a50-a1b3-b7dccc52fae7/volumes/kubernetes.io~projected/kube-api-access-km84r DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f59bf5f-da15-4174-a5be-b843478f0191/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1297e6ef-5035-497c-9777-c6b4448bceba/volumes/kubernetes.io~projected/kube-api-access-f6x56 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b38ebb5a-bd63-4e8c-94df-c0a904503c4c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3809a62c76782d05a5ab1b565d2b39220002200918ebeebac00247553e17b9f8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1e328bc-df61-43b5-a5cf-12a2389903a6/volumes/kubernetes.io~projected/kube-api-access-wwlfc DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce53bbfb2bf87bb9146509806c20754fffd7ff4bf7ca58db50ee758ac749a479/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7/volumes/kubernetes.io~projected/kube-api-access-8v7sm DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b253487198586d7fe47bf2bad1a818744ac7de28b2a03b8843a2637d06345e4/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c72eeb1-17c0-4dd4-99c3-2509ba36b668/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1b27776-d610-4efa-9f8e-e08f0161a07f/volumes/kubernetes.io~projected/kube-api-access-prj9t DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9c1e13d-3f76-4884-af1a-956dc34fd1d6/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~projected/kube-api-access-fdfnn DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~projected/kube-api-access-4b9kt DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca557849d78f9ca1965ca46ec0e7e6d7a7174dcdd63d56f448038f9d1e1951e8/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7/volumes/kubernetes.io~projected/kube-api-access-s5q49 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac149f85-b47b-4fb7-8ad7-3f985129f292/volumes/kubernetes.io~projected/kube-api-access-7k8p8 DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50011131-90de-4df2-ba1e-28b2a919dbf0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b0ae648-a85e-4a86-98f1-db2e9600c67a/volumes/kubernetes.io~projected/kube-api-access-knvdl DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91815a3a-773d-42b7-8777-d52bb36e0c8e/volumes/kubernetes.io~projected/kube-api-access-8xd5w DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c72eeb1-17c0-4dd4-99c3-2509ba36b668/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9fe381a8-6a81-4e21-96c6-715ecd22ff45/volumes/kubernetes.io~projected/kube-api-access-fdj6v DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ec856c2-103c-45fc-af43-bea87251934d/volumes/kubernetes.io~projected/kube-api-access-b92pj DeviceMajor:0 DeviceMinor:347 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d07a0e8ed908aae0ee04fa0954af3ec07c9f6d0f461e0ee4376735304279ee35/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c72eeb1-17c0-4dd4-99c3-2509ba36b668/volumes/kubernetes.io~projected/kube-api-access-xp6kc DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50011131-90de-4df2-ba1e-28b2a919dbf0/volumes/kubernetes.io~projected/kube-api-access-gtnkn DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/531aa693e7a74bc14f4db1b4297bb59e2f1f6e8cee45f437058dd7517a9a06e4/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d38d9fda59af58fef3810194e579b4f6d68ea4c7405dfc86ba96eb8d913fb00/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b38ebb5a-bd63-4e8c-94df-c0a904503c4c/volumes/kubernetes.io~projected/kube-api-access-pqzf2 DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9c1e13d-3f76-4884-af1a-956dc34fd1d6/volumes/kubernetes.io~projected/kube-api-access-v79fc DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/facc7d3c43fa65af9418b01f85949f531d104e9625d6ddc915228cbcf4362306/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:531aa693e7a74bc MacAddress:82:66:2a:8a:d6:93 Speed:10000 Mtu:8901} {Name:82dcbd39451e591 MacAddress:de:bc:0d:8e:10:bd Speed:10000 Mtu:8901} {Name:85128312e1bbe81 MacAddress:16:0b:54:57:7c:a6 Speed:10000 Mtu:8901} {Name:8cb1a8dd7005964 MacAddress:56:45:ee:e2:2e:17 Speed:10000 Mtu:8901} {Name:a8947f9b17cb096 MacAddress:5e:7d:b4:4d:bf:e5 Speed:10000 Mtu:8901} {Name:bbc6dc8dea02d76 MacAddress:16:8e:02:56:1e:cc Speed:10000 Mtu:8901} {Name:be1f38ab2368512 MacAddress:fa:f8:5a:3c:dd:9c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ee:ab:5e:a3 Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:48:64:c2:da:9e Speed:0 Mtu:8901} {Name:ca557849d78f9ca MacAddress:c6:5c:9f:e5:93:46 Speed:10000 Mtu:8901} {Name:cbf00387d7f8b5f MacAddress:7a:22:4f:c2:44:3d Speed:10000 Mtu:8901} {Name:ce53bbfb2bf87bb MacAddress:da:d8:bc:9a:56:e0 Speed:10000 Mtu:8901} {Name:dabb61cab0ada53 MacAddress:96:b7:dd:e4:06:64 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ee:ab:5e:a3 Speed:0 Mtu:9001} {Name:f1000233d40b379 MacAddress:46:6f:7f:84:28:5c Speed:10000 Mtu:8901} {Name:f2e931f8dde7769 MacAddress:36:2c:b4:1a:6d:da Speed:10000 Mtu:8901} {Name:f7ec2ee3429cb36 MacAddress:92:00:7d:b7:17:de Speed:10000 Mtu:8901} {Name:facc7d3c43fa65a MacAddress:fa:a3:73:ad:1c:dc Speed:10000 Mtu:8901} {Name:fc1ea5cc45e4f1d MacAddress:c2:53:47:2a:08:76 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:66:26:7a:00:79:22 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b6:a3:7a:46:29:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532071    5006 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532125    5006 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532438    5006 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532465    5006 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-104.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532604    5006 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532611    5006 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532622    5006 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532641    5006 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532676    5006 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532841    5006 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532949    5006 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532984    5006 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.532994    5006 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 19:16:21.532994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.533004    5006 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 19:16:21.533696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.533014    5006 kubelet.go:405] "Adding apiserver pod source"
Oct 31 19:16:21.533696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.533022    5006 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 19:16:21.533757 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.533737    5006 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 19:16:21.533757 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.533747    5006 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:16:21.534329 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.534311    5006 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 19:16:21.549566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.549542    5006 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 19:16:21.549566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.549559    5006 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:16:21.549745 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.549693    5006 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 19:16:21.549905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.549893    5006 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 19:16:21.549955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.549923    5006 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 19:16:21.550039 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550029    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 19:16:21.550080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550043    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 19:16:21.550080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550049    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 19:16:21.550080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550055    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 19:16:21.550080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550061    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 19:16:21.550080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550068    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 19:16:21.550080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550073    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 19:16:21.550080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550081    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 19:16:21.550296 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550092    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 19:16:21.550296 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550100    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 19:16:21.550296 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550111    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 19:16:21.550296 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550122    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 19:16:21.550296 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550148    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 19:16:21.550296 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.550158    5006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 19:16:21.553927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.553907    5006 server.go:1263] "Started kubelet"
Oct 31 19:16:21.554048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.553982    5006 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 19:16:21.554048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.554008    5006 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 19:16:21.554161 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.554085    5006 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 19:16:21.554518 ip-10-0-128-104 systemd[1]: Started Kubernetes Kubelet.
Oct 31 19:16:21.555234 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.555017    5006 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 19:16:21.559333 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.559303    5006 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 19:16:21.559795 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.559513    5006 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 19:16:21.559795 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.559527    5006 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 19:16:21.559795 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.559611    5006 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 19:16:21.559991 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:21.559865    5006 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:16:21.560391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.560367    5006 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 19:16:21.560544 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.560428    5006 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:10:40 +0000 UTC" deadline="2028-03-28 18:10:39.150417091 +0000 UTC"
Oct 31 19:16:21.560544 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.560468    5006 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12334h54m17.589952903s"
Oct 31 19:16:21.561917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.561895    5006 factory.go:55] Registering systemd factory
Oct 31 19:16:21.561917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.561918    5006 factory.go:223] Registration of the systemd container factory successfully
Oct 31 19:16:21.562346 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.562315    5006 factory.go:153] Registering CRI-O factory
Oct 31 19:16:21.562604 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.562449    5006 factory.go:223] Registration of the crio container factory successfully
Oct 31 19:16:21.562604 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.562512    5006 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 19:16:21.562604 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.562530    5006 factory.go:103] Registering Raw factory
Oct 31 19:16:21.562604 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.562545    5006 manager.go:1199] Started watching for new ooms in manager
Oct 31 19:16:21.562841 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.562318    5006 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 19:16:21.563222 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563202    5006 manager.go:319] Starting recovery of all containers
Oct 31 19:16:21.563715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563632    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50011131-90de-4df2-ba1e-28b2a919dbf0" volumeName="kubernetes.io/secret/50011131-90de-4df2-ba1e-28b2a919dbf0-cert" seLinuxMountContext=""
Oct 31 19:16:21.563799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563719    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f1221c-9c54-4098-a301-ece68db990eb" volumeName="kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-bound-sa-token" seLinuxMountContext=""
Oct 31 19:16:21.563799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563735    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae953c47-79ec-4415-9f33-d4fbb364d6c6" volumeName="kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-bound-sa-token" seLinuxMountContext=""
Oct 31 19:16:21.563799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563748    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0eaee38-fd4e-489a-a854-39245cd595dc" volumeName="kubernetes.io/projected/c0eaee38-fd4e-489a-a854-39245cd595dc-kube-api-access-p8qfj" seLinuxMountContext=""
Oct 31 19:16:21.563799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563762    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f59bf5f-da15-4174-a5be-b843478f0191" volumeName="kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 19:16:21.563799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563776    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="425443c8-69c7-464e-9f85-0578ff4effdc" volumeName="kubernetes.io/projected/425443c8-69c7-464e-9f85-0578ff4effdc-kube-api-access-w7f9m" seLinuxMountContext=""
Oct 31 19:16:21.563799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563790    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae953c47-79ec-4415-9f33-d4fbb364d6c6" volumeName="kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-trusted-ca" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563802    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1e328bc-df61-43b5-a5cf-12a2389903a6" volumeName="kubernetes.io/configmap/b1e328bc-df61-43b5-a5cf-12a2389903a6-telemetry-config" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563818    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b0ae648-a85e-4a86-98f1-db2e9600c67a" volumeName="kubernetes.io/projected/1b0ae648-a85e-4a86-98f1-db2e9600c67a-kube-api-access-knvdl" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563832    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c72eeb1-17c0-4dd4-99c3-2509ba36b668" volumeName="kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-env-overrides" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563844    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f59bf5f-da15-4174-a5be-b843478f0191" volumeName="kubernetes.io/projected/1f59bf5f-da15-4174-a5be-b843478f0191-kube-api-access-792p5" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563859    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="425443c8-69c7-464e-9f85-0578ff4effdc" volumeName="kubernetes.io/configmap/425443c8-69c7-464e-9f85-0578ff4effdc-config-volume" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563872    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613da58f-accf-4a50-a1b3-b7dccc52fae7" volumeName="kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-cni-binary-copy" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563885    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91815a3a-773d-42b7-8777-d52bb36e0c8e" volumeName="kubernetes.io/projected/91815a3a-773d-42b7-8777-d52bb36e0c8e-kube-api-access-8xd5w" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563901    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4" volumeName="kubernetes.io/projected/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-kube-api-access-g7wg7" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563913    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae953c47-79ec-4415-9f33-d4fbb364d6c6" volumeName="kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563932    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec856c2-103c-45fc-af43-bea87251934d" volumeName="kubernetes.io/projected/0ec856c2-103c-45fc-af43-bea87251934d-kube-api-access-b92pj" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563962    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fe381a8-6a81-4e21-96c6-715ecd22ff45" volumeName="kubernetes.io/projected/9fe381a8-6a81-4e21-96c6-715ecd22ff45-kube-api-access-fdj6v" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.563976    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4" volumeName="kubernetes.io/configmap/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-iptables-alerter-script" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564004    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b38ebb5a-bd63-4e8c-94df-c0a904503c4c" volumeName="kubernetes.io/configmap/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-config" seLinuxMountContext=""
Oct 31 19:16:21.564015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564017    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed0af9e6-dea0-442a-b9a5-50adacf2f3e7" volumeName="kubernetes.io/projected/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-api-access-8v7sm" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564030    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f59bf5f-da15-4174-a5be-b843478f0191" volumeName="kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-service-ca-bundle" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564043    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25311b12-0544-48e2-8b00-f56aff90596d" volumeName="kubernetes.io/projected/25311b12-0544-48e2-8b00-f56aff90596d-kube-api-access-sfd9g" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564058    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e36ee6a-f51a-4182-8bfb-68da77946954" volumeName="kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-binary-copy" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564071    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e36ee6a-f51a-4182-8bfb-68da77946954" volumeName="kubernetes.io/projected/4e36ee6a-f51a-4182-8bfb-68da77946954-kube-api-access-wcwv2" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564083    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50011131-90de-4df2-ba1e-28b2a919dbf0" volumeName="kubernetes.io/projected/50011131-90de-4df2-ba1e-28b2a919dbf0-kube-api-access-gtnkn" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564097    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f1221c-9c54-4098-a301-ece68db990eb" volumeName="kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-trusted-ca" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564109    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f1221c-9c54-4098-a301-ece68db990eb" volumeName="kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-kube-api-access-4b9kt" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564127    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78ed8e27-2cd2-43e5-8ee1-23042619743e" volumeName="kubernetes.io/projected/78ed8e27-2cd2-43e5-8ee1-23042619743e-kube-api-access-v9dh8" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564141    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c39943b-0a43-4184-b616-dda8d9f8545e" volumeName="kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564154    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f1221c-9c54-4098-a301-ece68db990eb" volumeName="kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-registry-certificates" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564167    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3897285-e4e6-45da-bf32-fcadd1f9b506" volumeName="kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-config" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564179    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3897285-e4e6-45da-bf32-fcadd1f9b506" volumeName="kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-trusted-ca" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564193    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae953c47-79ec-4415-9f33-d4fbb364d6c6" volumeName="kubernetes.io/empty-dir/ae953c47-79ec-4415-9f33-d4fbb364d6c6-ca-trust-extracted" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564216    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae953c47-79ec-4415-9f33-d4fbb364d6c6" volumeName="kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-installation-pull-secrets" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564229    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b38ebb5a-bd63-4e8c-94df-c0a904503c4c" volumeName="kubernetes.io/projected/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-kube-api-access-pqzf2" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564241    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1b27776-d610-4efa-9f8e-e08f0161a07f" volumeName="kubernetes.io/projected/c1b27776-d610-4efa-9f8e-e08f0161a07f-kube-api-access-prj9t" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564256    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f1221c-9c54-4098-a301-ece68db990eb" volumeName="kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-installation-pull-secrets" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564268    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b37ff40c-3db6-4467-aeaa-c23407b5fce5" volumeName="kubernetes.io/projected/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kube-api-access-4tlwl" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564280    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c70d21f3-352d-45fc-853c-a74f690d1898" volumeName="kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564292    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9c1e13d-3f76-4884-af1a-956dc34fd1d6" volumeName="kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-stats-auth" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564307    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd" volumeName="kubernetes.io/projected/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-kube-api-access-6d6vs" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564320    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f59bf5f-da15-4174-a5be-b843478f0191" volumeName="kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-tmp" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564334    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613da58f-accf-4a50-a1b3-b7dccc52fae7" volumeName="kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-daemon-config" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564346    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f1221c-9c54-4098-a301-ece68db990eb" volumeName="kubernetes.io/empty-dir/74f1221c-9c54-4098-a301-ece68db990eb-ca-trust-extracted" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564359    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1b27776-d610-4efa-9f8e-e08f0161a07f" volumeName="kubernetes.io/configmap/c1b27776-d610-4efa-9f8e-e08f0161a07f-config" seLinuxMountContext=""
Oct 31 19:16:21.564367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564371    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed0af9e6-dea0-442a-b9a5-50adacf2f3e7" volumeName="kubernetes.io/empty-dir/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-data-volume" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564385    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1297e6ef-5035-497c-9777-c6b4448bceba" volumeName="kubernetes.io/projected/1297e6ef-5035-497c-9777-c6b4448bceba-kube-api-access-f6x56" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564398    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b0ae648-a85e-4a86-98f1-db2e9600c67a" volumeName="kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-service-ca-bundle" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564412    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c72eeb1-17c0-4dd4-99c3-2509ba36b668" volumeName="kubernetes.io/secret/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564436    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="291cf47e-f759-4489-abb5-1bafffff7b25" volumeName="kubernetes.io/projected/291cf47e-f759-4489-abb5-1bafffff7b25-kube-api-access-nqdqq" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564449    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e36ee6a-f51a-4182-8bfb-68da77946954" volumeName="kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564461    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac149f85-b47b-4fb7-8ad7-3f985129f292" volumeName="kubernetes.io/projected/ac149f85-b47b-4fb7-8ad7-3f985129f292-kube-api-access-7k8p8" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564475    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c72eeb1-17c0-4dd4-99c3-2509ba36b668" volumeName="kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-config" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564488    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c72eeb1-17c0-4dd4-99c3-2509ba36b668" volumeName="kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564501    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="425443c8-69c7-464e-9f85-0578ff4effdc" volumeName="kubernetes.io/empty-dir/425443c8-69c7-464e-9f85-0578ff4effdc-tmp-dir" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564523    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42d6c1fe-fc02-4e60-9d54-3b66eb513ed7" volumeName="kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-tuned" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564536    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e36ee6a-f51a-4182-8bfb-68da77946954" volumeName="kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564549    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b38ebb5a-bd63-4e8c-94df-c0a904503c4c" volumeName="kubernetes.io/secret/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-serving-cert" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564561    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40" volumeName="kubernetes.io/configmap/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-serviceca" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564574    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25311b12-0544-48e2-8b00-f56aff90596d" volumeName="kubernetes.io/configmap/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-config" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564588    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73377bc4-2624-4ed6-a5aa-30228b89cbfb" volumeName="kubernetes.io/projected/73377bc4-2624-4ed6-a5aa-30228b89cbfb-kube-api-access-ls8vr" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564600    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3897285-e4e6-45da-bf32-fcadd1f9b506" volumeName="kubernetes.io/projected/a3897285-e4e6-45da-bf32-fcadd1f9b506-kube-api-access-qjmw8" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564613    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b05731a8-35c3-4afa-97fa-6d3db50533b3" volumeName="kubernetes.io/secret/b05731a8-35c3-4afa-97fa-6d3db50533b3-agent-certs" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564624    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40" volumeName="kubernetes.io/projected/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-kube-api-access-njvjg" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564662    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1b27776-d610-4efa-9f8e-e08f0161a07f" volumeName="kubernetes.io/secret/c1b27776-d610-4efa-9f8e-e08f0161a07f-serving-cert" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564675    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9c1e13d-3f76-4884-af1a-956dc34fd1d6" volumeName="kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-default-certificate" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564687    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd" volumeName="kubernetes.io/configmap/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-cabundle" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564699    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f59bf5f-da15-4174-a5be-b843478f0191" volumeName="kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-snapshots" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564711    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1e328bc-df61-43b5-a5cf-12a2389903a6" volumeName="kubernetes.io/projected/b1e328bc-df61-43b5-a5cf-12a2389903a6-kube-api-access-wwlfc" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564726    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38ce8722-6108-4502-b83d-0a29a46f88b0" volumeName="kubernetes.io/configmap/38ce8722-6108-4502-b83d-0a29a46f88b0-nginx-conf" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564739    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f1221c-9c54-4098-a301-ece68db990eb" volumeName="kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564751    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fe381a8-6a81-4e21-96c6-715ecd22ff45" volumeName="kubernetes.io/empty-dir/9fe381a8-6a81-4e21-96c6-715ecd22ff45-tmp-dir" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564763    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3897285-e4e6-45da-bf32-fcadd1f9b506" volumeName="kubernetes.io/secret/a3897285-e4e6-45da-bf32-fcadd1f9b506-serving-cert" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564775    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae953c47-79ec-4415-9f33-d4fbb364d6c6" volumeName="kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-certificates" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564787    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b05731a8-35c3-4afa-97fa-6d3db50533b3" volumeName="kubernetes.io/configmap/b05731a8-35c3-4afa-97fa-6d3db50533b3-konnectivity-ca" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564799    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b0ae648-a85e-4a86-98f1-db2e9600c67a" volumeName="kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564813    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c72eeb1-17c0-4dd4-99c3-2509ba36b668" volumeName="kubernetes.io/projected/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-kube-api-access-xp6kc" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564825    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42d6c1fe-fc02-4e60-9d54-3b66eb513ed7" volumeName="kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-tmp" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564838    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613da58f-accf-4a50-a1b3-b7dccc52fae7" volumeName="kubernetes.io/projected/613da58f-accf-4a50-a1b3-b7dccc52fae7-kube-api-access-km84r" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564852    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd" volumeName="kubernetes.io/secret/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-key" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564871    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f59bf5f-da15-4174-a5be-b843478f0191" volumeName="kubernetes.io/secret/1f59bf5f-da15-4174-a5be-b843478f0191-serving-cert" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564884    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42d6c1fe-fc02-4e60-9d54-3b66eb513ed7" volumeName="kubernetes.io/projected/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-kube-api-access-s5q49" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564897    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae953c47-79ec-4415-9f33-d4fbb364d6c6" volumeName="kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-kube-api-access-fdfnn" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564908    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9c1e13d-3f76-4884-af1a-956dc34fd1d6" volumeName="kubernetes.io/projected/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-kube-api-access-v79fc" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564921    5006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b0ae648-a85e-4a86-98f1-db2e9600c67a" volumeName="kubernetes.io/empty-dir/1b0ae648-a85e-4a86-98f1-db2e9600c67a-archives-path" seLinuxMountContext=""
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564933    5006 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 19:16:21.565107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.564941    5006 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 19:16:21.569955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.568258    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/1.log"
Oct 31 19:16:21.573472 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.573449    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/1.log"
Oct 31 19:16:21.588771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.588725    5006 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 19:16:21.590173 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.590148    5006 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 19:16:21.590173 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.590170    5006 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 19:16:21.590332 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.590189    5006 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 19:16:21.590332 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:21.590234    5006 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 19:16:21.609690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.609631    5006 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="b37eb1661ffba7ce4abe7e1f1f6607c65df035a2c0d6c1202ac3550fbc358e18" exitCode=0
Oct 31 19:16:21.609690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.609684    5006 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="40fb9dddcdf07daf1f08629419fcf039326eb81c3d51504c63649da51c0ecfb2" exitCode=0
Oct 31 19:16:21.609690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.609697    5006 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="41c9331b1b906a9a6a359852fd9a49065dac681151065120e052bd2efc154f13" exitCode=0
Oct 31 19:16:21.609913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.609707    5006 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="b18580c3130edaa8c2f6810571c899bb5b63ffb9445d4681e1f94ab1bebb0013" exitCode=0
Oct 31 19:16:21.609913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.609713    5006 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="80cea43c8f9da61662e0d6b92bedcb2dc6231282a84820b4c9f6fc6c5af28104" exitCode=0
Oct 31 19:16:21.609913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.609719    5006 generic.go:356] "Generic (PLEG): container finished" podID="4e36ee6a-f51a-4182-8bfb-68da77946954" containerID="18a41580961d6137b1c98cea9e6f03f4fc7996686fc56abcf5a747653755e329" exitCode=0
Oct 31 19:16:21.620207 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.620125    5006 generic.go:356] "Generic (PLEG): container finished" podID="f83e5200bcb2ab7f96a88eeb2451b2a6" containerID="15b044bd6ec1b426d8e419feec768a4dcd0858e673ffe50abd2db2bf71a8478a" exitCode=0
Oct 31 19:16:21.638217 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.638171    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/1.log"
Oct 31 19:16:21.638374 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.638227    5006 generic.go:356] "Generic (PLEG): container finished" podID="a3897285-e4e6-45da-bf32-fcadd1f9b506" containerID="f9ab0191185107cec3cb6699586632b12dd63fd0ad52afbc12b5b9953e787211" exitCode=255
Oct 31 19:16:21.660018 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:21.659989    5006 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:16:21.669193 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.669159    5006 manager.go:324] Recovery completed
Oct 31 19:16:21.687631 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.687604    5006 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:16:21.688956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.688935    5006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:16:21.689087 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.688967    5006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:16:21.689087 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.688979    5006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:16:21.690673 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:21.690622    5006 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 19:16:21.691720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691704    5006 cpu_manager.go:226] "Starting" policy="none"
Oct 31 19:16:21.691720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691719    5006 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 19:16:21.691805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691728    5006 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:16:21.691805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691739    5006 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 19:16:21.691897 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691879    5006 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 19:16:21.691926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691890    5006 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 19:16:21.691926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691910    5006 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 19:16:21.691926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691919    5006 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 19:16:21.691926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691926    5006 policy_none.go:50] "Start"
Oct 31 19:16:21.692027 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691939    5006 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 19:16:21.692027 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.691951    5006 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 19:16:21.692136 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.692124    5006 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 19:16:21.692169 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.692136    5006 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 19:16:21.692169 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.692143    5006 policy_none.go:44] "Start"
Oct 31 19:16:21.692862 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.692837    5006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:16:21.692948 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.692871    5006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:16:21.692948 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.692885    5006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:16:21.697351 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.697315    5006 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 19:16:21.706837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.706814    5006 manager.go:341] "Starting Device Plugin manager"
Oct 31 19:16:21.706975 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:21.706863    5006 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 19:16:21.706975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.706878    5006 server.go:87] "Starting device plugin registration server"
Oct 31 19:16:21.707320 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.707290    5006 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 19:16:21.707408 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.707311    5006 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 19:16:21.708053 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.708035    5006 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 19:16:21.708384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.708143    5006 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 19:16:21.708384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.708182    5006 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 19:16:21.708384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.708190    5006 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 19:16:21.709428 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.708258    5006 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:16:21.708169819Z","UUID":"28cfd3e1-a82c-40b1-9c1b-036eea0d7c61","Handler":null,"Name":"","Endpoint":""}
Oct 31 19:16:21.710455 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:21.710427    5006 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 19:16:21.711746 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:21.711720    5006 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-104.us-west-2.compute.internal\" not found"
Oct 31 19:16:21.712233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.712212    5006 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 19:16:21.712329 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.712246    5006 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 19:16:21.743048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.743017    5006 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 19:16:21.743367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.743339    5006 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:16:21.744028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.743997    5006 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:16:21.744194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.744174    5006 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:16:21.794631 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.794598    5006 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:16:21.807970 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.807933    5006 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:16:21.809316 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.809293    5006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:16:21.809447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.809329    5006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:16:21.809447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.809343    5006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-104.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:16:21.809447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.809396    5006 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:21.891685 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.891532    5006 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"]
Oct 31 19:16:21.892372 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.892352    5006 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:21.892484 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.892336    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal" event={"ID":"f83e5200bcb2ab7f96a88eeb2451b2a6","Type":"ContainerStarted","Data":"1473f7eeee0a5232f33be8509b7ade7b7be90126d5af393ad8d2f0bda37316e5"}
Oct 31 19:16:21.892598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.892493    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal" event={"ID":"f83e5200bcb2ab7f96a88eeb2451b2a6","Type":"ContainerDied","Data":"15b044bd6ec1b426d8e419feec768a4dcd0858e673ffe50abd2db2bf71a8478a"}
Oct 31 19:16:21.892598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.892522    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal" event={"ID":"f83e5200bcb2ab7f96a88eeb2451b2a6","Type":"ContainerStarted","Data":"ac545f00e57d82c65d2fbd08eb776c5127df5595567899a4dd1c433595badb75"}
Oct 31 19:16:21.892598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.892534    5006 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:21.892598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.892560    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal" event={"ID":"70cf5f03a5aad2f217175f2d65705fc5","Type":"ContainerStarted","Data":"05345beb120d205be3ea9e0d4a7537b36155978c2dc9f991271f2408ed9e94ab"}
Oct 31 19:16:21.892598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.892580    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal" event={"ID":"70cf5f03a5aad2f217175f2d65705fc5","Type":"ContainerStarted","Data":"d07a0e8ed908aae0ee04fa0954af3ec07c9f6d0f461e0ee4376735304279ee35"}
Oct 31 19:16:21.918991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.918956    5006 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:21.919145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.919063    5006 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:21.957374 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.957339    5006 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:16:21.957625 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:21.957399    5006 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:21.962731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:21.959950    5006 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:16:21.962731 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:21.960035    5006 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.043879 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.043777    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.044071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.043901    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.044071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.043919    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70cf5f03a5aad2f217175f2d65705fc5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"70cf5f03a5aad2f217175f2d65705fc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.145036 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.144929    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.145036 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.144987    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.145036 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.145013    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70cf5f03a5aad2f217175f2d65705fc5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"70cf5f03a5aad2f217175f2d65705fc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.145240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.145047    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.145240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.145064    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70cf5f03a5aad2f217175f2d65705fc5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"70cf5f03a5aad2f217175f2d65705fc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.145240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.145106    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83e5200bcb2ab7f96a88eeb2451b2a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" (UID: \"f83e5200bcb2ab7f96a88eeb2451b2a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.533462 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.533375    5006 apiserver.go:52] "Watching apiserver"
Oct 31 19:16:22.668447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.668419    5006 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.668627 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.668501    5006 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.688577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.688545    5006 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 19:16:22.689325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.689294    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-kttjn","openshift-dns/dns-default-tcmmx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82","kube-system/kas-connection-checker-685895c7c9-wq4hz","kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal","openshift-ingress-canary/ingress-canary-62bwf","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb","openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l","kube-system/global-pull-secret-syncer-vs8zn","openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l","kube-system/kas-connection-checker-685895c7c9-vmhw6","openshift-multus/multus-additional-cni-plugins-gbmm4","openshift-insights/insights-runtime-extractor-fnscp","openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc","openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v","kube-system/kas-connection-checker-685895c7c9-mjspt","openshift-network-console/networking-console-plugin-86c478bb88-jcp2j","openshift-ovn-kubernetes/ovnkube-node-xrnqv","openshift-cluster-node-tuning-operator/tuned-5psz6","openshift-dns/node-resolver-8gxf9","openshift-image-registry/image-registry-55c7865b76-fzmxv","openshift-network-operator/iptables-alerter-vg9dn","openshift-service-ca/service-ca-5fffc97bdf-hhtc4","openshift-multus/multus-4njcn","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst","openshift-multus/network-metrics-daemon-nptg8","openshift-insights/periodic-gathering-wl28r-zphd5","openshift-network-diagnostics/network-check-target-g5rjx","openshift-image-registry/node-ca-t5726","openshift-image-registry/image-registry-65f799dcd5-gmv4v","openshift-console-operator/console-operator-7cd56b8dbc-nztjc","openshift-ingress/router-default-ddc6dfd67-fk8lv","openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg","openshift-insights/insights-operator-7c4986d449-gdrr2"]
Oct 31 19:16:22.694331 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.692085    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:22.695305 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.695278    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.695678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.695661    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:22.696370 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.696350    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.696781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.696766    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:22.701598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.701574    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:22.702871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.702852    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.704016 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.703998    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:22.705038 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.705020    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.730532 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.730496    5006 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:16:22.730714 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.730563    5006 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.730984 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.730971    5006 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:16:22.731016 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.731006    5006 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-104.us-west-2.compute.internal"
Oct 31 19:16:22.743211 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.743180    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.743452 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.743433    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.743573 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.743546    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-b72kc\""
Oct 31 19:16:22.743756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.743576    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 19:16:22.743756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.743645    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 19:16:22.743756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.743678    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8n9ml\""
Oct 31 19:16:22.743756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.743730    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.744173 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.744153    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.744284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.744268    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.744335 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.744291    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 19:16:22.744335 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.744302    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 19:16:22.744473 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.744461    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.745299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.745281    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 19:16:22.745635 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.745619    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.745769 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.745706    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.745769 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.745739    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t6kdz\""
Oct 31 19:16:22.746373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.746352    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 19:16:22.746496 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.746423    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 19:16:22.746561 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.746524    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sqb42\""
Oct 31 19:16:22.746973 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.746943    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.747062 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.747031    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.747204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.747176    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 19:16:22.747388 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.747372    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.747388 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.747380    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 19:16:22.747666 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.747626    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.747747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.747730    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 19:16:22.747951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.747933    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ml5rl\""
Oct 31 19:16:22.761740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.761712    5006 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 19:16:22.795242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795162    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lwj7k\""
Oct 31 19:16:22.795242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795162    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.795242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795176    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.795597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795172    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.795597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795206    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.795597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795261    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.795597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795294    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-llq5c\""
Oct 31 19:16:22.795597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795417    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 19:16:22.795597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795493    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 19:16:22.795934 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795918    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 19:16:22.795972 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.795959    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 19:16:22.796020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796001    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 19:16:22.796054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796024    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z8492\""
Oct 31 19:16:22.796102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796065    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgqq8\""
Oct 31 19:16:22.796102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796014    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlrbg\""
Oct 31 19:16:22.796102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796087    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.796233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796109    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 19:16:22.796233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796125    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qh5p\""
Oct 31 19:16:22.796233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796154    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 19:16:22.796233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796204    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.796233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796220    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796249    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796261    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796274    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796299    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796304    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796326    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796320    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796262    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796361    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796377    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796327    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796260    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796407    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796413    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796442    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 19:16:22.796460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796454    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796488    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796603    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796618    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796622    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796628    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796677    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796683    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796710    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796732    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796763    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796770    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796791    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796817    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796861    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796908    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796948    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796970    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.796994    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 19:16:22.797096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.797008    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 19:16:22.823353 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.823324    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 19:16:22.823353 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.823333    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dhmj8\""
Oct 31 19:16:22.823517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.823388    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 19:16:22.823517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.823475    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-db5zp\""
Oct 31 19:16:22.823578 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.823530    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 19:16:22.823768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.823754    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 19:16:22.824047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824032    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 19:16:22.824047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824040    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 19:16:22.824134 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824042    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 19:16:22.824174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824130    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 19:16:22.824174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824146    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rdjmx\""
Oct 31 19:16:22.824269 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824255    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgzhm\""
Oct 31 19:16:22.824335 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824281    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 19:16:22.824335 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824319    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 19:16:22.824405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824334    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mcs56\""
Oct 31 19:16:22.824405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824392    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-htdcp\""
Oct 31 19:16:22.824487 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824435    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 19:16:22.824487 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824444    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5crdh\""
Oct 31 19:16:22.824487 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824446    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d7wnz\""
Oct 31 19:16:22.824487 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824481    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tqdrb\""
Oct 31 19:16:22.824684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824499    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 19:16:22.824684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824512    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4b89l\""
Oct 31 19:16:22.824684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824512    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2kswh\""
Oct 31 19:16:22.824684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824546    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zsjk9\""
Oct 31 19:16:22.824684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824588    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 19:16:22.824684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824589    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tgkwh\""
Oct 31 19:16:22.824684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824622    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 19:16:22.824684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824683    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-s8g85\""
Oct 31 19:16:22.824927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824729    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4ftk8\""
Oct 31 19:16:22.824927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824751    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5n42z\""
Oct 31 19:16:22.824927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.824774    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vgfpk\""
Oct 31 19:16:22.849099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849065    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-service-ca-bundle\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:22.849099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849103    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-ovn\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.849278 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849128    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dh8\" (UniqueName: \"kubernetes.io/projected/78ed8e27-2cd2-43e5-8ee1-23042619743e-kube-api-access-v9dh8\") pod \"kas-connection-checker-685895c7c9-mjspt\" (UID: \"78ed8e27-2cd2-43e5-8ee1-23042619743e\") " pod="kube-system/kas-connection-checker-685895c7c9-mjspt"
Oct 31 19:16:22.849278 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849192    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-iptables-alerter-script\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:16:22.849278 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849234    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-host\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.849278 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849263    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp6kc\" (UniqueName: \"kubernetes.io/projected/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-kube-api-access-xp6kc\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.849473 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849450    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-installation-pull-secrets\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.849537 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849491    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b05731a8-35c3-4afa-97fa-6d3db50533b3-agent-certs\") pod \"konnectivity-agent-kttjn\" (UID: \"b05731a8-35c3-4afa-97fa-6d3db50533b3\") " pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:16:22.849537 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849530    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls8vr\" (UniqueName: \"kubernetes.io/projected/73377bc4-2624-4ed6-a5aa-30228b89cbfb-kube-api-access-ls8vr\") pod \"kas-connection-checker-685895c7c9-wq4hz\" (UID: \"73377bc4-2624-4ed6-a5aa-30228b89cbfb\") " pod="kube-system/kas-connection-checker-685895c7c9-wq4hz"
Oct 31 19:16:22.849637 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849563    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:22.849637 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849593    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-daemon-config\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.849753 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849640    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-iptables-alerter-script\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:16:22.849753 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849728    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-installation-pull-secrets\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.849857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849781    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-trusted-ca\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.849857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849816    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-service-ca-bundle\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:22.849857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849819    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-node-log\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.849985 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.849916    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-daemon-config\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.850023 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850002    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-cabundle\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:22.850054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850025    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8p8\" (UniqueName: \"kubernetes.io/projected/ac149f85-b47b-4fb7-8ad7-3f985129f292-kube-api-access-7k8p8\") pod \"network-check-source-6f78d59f99-zkvsg\" (UID: \"ac149f85-b47b-4fb7-8ad7-3f985129f292\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg"
Oct 31 19:16:22.850054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850029    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b05731a8-35c3-4afa-97fa-6d3db50533b3-agent-certs\") pod \"konnectivity-agent-kttjn\" (UID: \"b05731a8-35c3-4afa-97fa-6d3db50533b3\") " pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:16:22.850054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850044    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-device-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.850161 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850061    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50011131-90de-4df2-ba1e-28b2a919dbf0-cert\") pod \"ingress-canary-62bwf\" (UID: \"50011131-90de-4df2-ba1e-28b2a919dbf0\") " pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:22.850161 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850114    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-serviceca\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:16:22.850161 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850141    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-trusted-ca\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.850355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850221    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.850355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850327    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-certificates\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.850455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850403    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-serviceca\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:16:22.850455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850407    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-cabundle\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:22.850455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850422    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-run\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.850455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850451    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-sys\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.850667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850458    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-certificates\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.850667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850461    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50011131-90de-4df2-ba1e-28b2a919dbf0-cert\") pod \"ingress-canary-62bwf\" (UID: \"50011131-90de-4df2-ba1e-28b2a919dbf0\") " pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:22.850667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850491    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xd5w\" (UniqueName: \"kubernetes.io/projected/91815a3a-773d-42b7-8777-d52bb36e0c8e-kube-api-access-8xd5w\") pod \"migrator-66df5d496b-vzd5v\" (UID: \"91815a3a-773d-42b7-8777-d52bb36e0c8e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v"
Oct 31 19:16:22.850667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850546    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-image-registry-private-configuration\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.850667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850568    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1b27776-d610-4efa-9f8e-e08f0161a07f-config\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:22.850854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850682    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-netns\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.850854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850716    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.850854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850750    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1b27776-d610-4efa-9f8e-e08f0161a07f-config\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:22.850854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850785    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-service-ca-bundle\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.850854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850791    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-image-registry-private-configuration\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.850854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850848    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.851019 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850918    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvjg\" (UniqueName: \"kubernetes.io/projected/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-kube-api-access-njvjg\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:16:22.851019 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.850943    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdqq\" (UniqueName: \"kubernetes.io/projected/291cf47e-f759-4489-abb5-1bafffff7b25-kube-api-access-nqdqq\") pod \"kas-connection-checker-685895c7c9-vmhw6\" (UID: \"291cf47e-f759-4489-abb5-1bafffff7b25\") " pod="kube-system/kas-connection-checker-685895c7c9-vmhw6"
Oct 31 19:16:22.851111 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851036    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-cnibin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.851160 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851117    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-kubernetes\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.851210 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851161    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knvdl\" (UniqueName: \"kubernetes.io/projected/1b0ae648-a85e-4a86-98f1-db2e9600c67a-kube-api-access-knvdl\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:22.851210 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851187    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-systemd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.851349 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851313    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-service-ca-bundle\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.851407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851351    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-var-lib-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.851407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851376    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-bound-sa-token\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.851407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851394    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-host-slash\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:16:22.851539 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851448    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:22.851539 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851471    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-system-cni-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.851539 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851492    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-modprobe-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.851708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851538    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-var-lib-kubelet\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.851708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851597    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.851708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851621    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fe381a8-6a81-4e21-96c6-715ecd22ff45-tmp-dir\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:16:22.851708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851662    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovn-node-metrics-cert\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.851708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851705    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fe381a8-6a81-4e21-96c6-715ecd22ff45-tmp-dir\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:16:22.851942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851706    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-trusted-ca\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.851942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851851    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79fc\" (UniqueName: \"kubernetes.io/projected/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-kube-api-access-v79fc\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.851942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.851895    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-trusted-ca\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:22.852133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852005    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovn-node-metrics-cert\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.852133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852043    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:22.852133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852111    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-trusted-ca\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.852133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852114    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-kubelet\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.852299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852162    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:22.852299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852193    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:22.852299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852214    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnkn\" (UniqueName: \"kubernetes.io/projected/50011131-90de-4df2-ba1e-28b2a919dbf0-kube-api-access-gtnkn\") pod \"ingress-canary-62bwf\" (UID: \"50011131-90de-4df2-ba1e-28b2a919dbf0\") " pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:22.852299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852246    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:22.852299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852277    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqzf2\" (UniqueName: \"kubernetes.io/projected/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-kube-api-access-pqzf2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:22.852299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852278    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-trusted-ca\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:22.852467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852390    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-socket-dir-parent\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.852467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852418    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:22.852467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852437    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-systemd\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.852552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852470    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:22.852552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852522    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prj9t\" (UniqueName: \"kubernetes.io/projected/c1b27776-d610-4efa-9f8e-e08f0161a07f-kube-api-access-prj9t\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:22.852627 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852610    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.852687 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852613    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:22.852720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852691    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.852720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852716    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-installation-pull-secrets\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.852815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852740    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-key\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:22.852815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852767    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-image-registry-private-configuration\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.852815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852788    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6x56\" (UniqueName: \"kubernetes.io/projected/1297e6ef-5035-497c-9777-c6b4448bceba-kube-api-access-f6x56\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:22.852815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852804    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-cni-binary-copy\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.853004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852832    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.853004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852857    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.853004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852880    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysconfig\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.853004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852905    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:22.853004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852930    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1b27776-d610-4efa-9f8e-e08f0161a07f-serving-cert\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:22.853004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852940    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-installation-pull-secrets\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.853004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.852954    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7wg7\" (UniqueName: \"kubernetes.io/projected/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-kube-api-access-g7wg7\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853005    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-crio-socket\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853011    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/613da58f-accf-4a50-a1b3-b7dccc52fae7-cni-binary-copy\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853026    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-signing-key\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853034    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fe381a8-6a81-4e21-96c6-715ecd22ff45-hosts-file\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853002    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-image-registry-private-configuration\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853069    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853076    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853110    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1b27776-d610-4efa-9f8e-e08f0161a07f-serving-cert\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853110    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-etc-selinux\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853141    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmw8\" (UniqueName: \"kubernetes.io/projected/a3897285-e4e6-45da-bf32-fcadd1f9b506-kube-api-access-qjmw8\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853160    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km84r\" (UniqueName: \"kubernetes.io/projected/613da58f-accf-4a50-a1b3-b7dccc52fae7-kube-api-access-km84r\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853176    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853193    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b92pj\" (UniqueName: \"kubernetes.io/projected/0ec856c2-103c-45fc-af43-bea87251934d-kube-api-access-b92pj\") pod \"migrator-66df5d496b-ldkvc\" (UID: \"0ec856c2-103c-45fc-af43-bea87251934d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853210    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdfnn\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-kube-api-access-fdfnn\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.853254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853238    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c39943b-0a43-4184-b616-dda8d9f8545e-original-pull-secret\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853321    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-bin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853356    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwv2\" (UniqueName: \"kubernetes.io/projected/4e36ee6a-f51a-4182-8bfb-68da77946954-kube-api-access-wcwv2\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853430    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1b0ae648-a85e-4a86-98f1-db2e9600c67a-archives-path\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853460    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853493    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853528    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1e328bc-df61-43b5-a5cf-12a2389903a6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853527    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1b0ae648-a85e-4a86-98f1-db2e9600c67a-archives-path\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853584    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-conf\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853607    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7f9m\" (UniqueName: \"kubernetes.io/projected/425443c8-69c7-464e-9f85-0578ff4effdc-kube-api-access-w7f9m\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853623    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-log-socket\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853667    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-netd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.853729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853715    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74f1221c-9c54-4098-a301-ece68db990eb-ca-trust-extracted\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853765    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1e328bc-df61-43b5-a5cf-12a2389903a6-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853818    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853847    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74f1221c-9c54-4098-a301-ece68db990eb-ca-trust-extracted\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853879    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.853988    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae953c47-79ec-4415-9f33-d4fbb364d6c6-ca-trust-extracted\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854020    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tlwl\" (UniqueName: \"kubernetes.io/projected/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kube-api-access-4tlwl\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854037    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdj6v\" (UniqueName: \"kubernetes.io/projected/9fe381a8-6a81-4e21-96c6-715ecd22ff45-kube-api-access-fdj6v\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854055    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-bound-sa-token\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854070    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6vs\" (UniqueName: \"kubernetes.io/projected/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-kube-api-access-6d6vs\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854088    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7sm\" (UniqueName: \"kubernetes.io/projected/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-api-access-8v7sm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854094    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae953c47-79ec-4415-9f33-d4fbb364d6c6-ca-trust-extracted\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854104    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854142    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59bf5f-da15-4174-a5be-b843478f0191-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854152    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-multus\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854178    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5q49\" (UniqueName: \"kubernetes.io/projected/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-kube-api-access-s5q49\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.854228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854213    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b05731a8-35c3-4afa-97fa-6d3db50533b3-konnectivity-ca\") pod \"konnectivity-agent-kttjn\" (UID: \"b05731a8-35c3-4afa-97fa-6d3db50533b3\") " pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854335    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-snapshots\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854388    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-k8s-cni-cncf-io\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854415    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-snapshots\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854424    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b05731a8-35c3-4afa-97fa-6d3db50533b3-konnectivity-ca\") pod \"konnectivity-agent-kttjn\" (UID: \"b05731a8-35c3-4afa-97fa-6d3db50533b3\") " pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854418    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-conf-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854463    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-cnibin\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854495    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-systemd-units\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854525    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f59bf5f-da15-4174-a5be-b843478f0191-serving-cert\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854552    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-stats-auth\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854584    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-system-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854610    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854640    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-script-lib\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854691    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f59bf5f-da15-4174-a5be-b843478f0191-serving-cert\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854729    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9kt\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-kube-api-access-4b9kt\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854763    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8qfj\" (UniqueName: \"kubernetes.io/projected/c0eaee38-fd4e-489a-a854-39245cd595dc-kube-api-access-p8qfj\") pod \"volume-data-source-validator-58c9f5f46d-24dst\" (UID: \"c0eaee38-fd4e-489a-a854-39245cd595dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854793    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854819    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-tuned\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854837    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-stats-auth\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854848    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-trusted-ca-bundle\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:22.854847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854836    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-script-lib\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854879    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-socket-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854881    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-tuned\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854904    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-data-volume\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854939    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854969    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/425443c8-69c7-464e-9f85-0578ff4effdc-config-volume\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.854996    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e36ee6a-f51a-4182-8bfb-68da77946954-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855002    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-data-volume\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855023    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-bin\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855059    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-config\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855097    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-792p5\" (UniqueName: \"kubernetes.io/projected/1f59bf5f-da15-4174-a5be-b843478f0191-kube-api-access-792p5\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855135    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855168    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfd9g\" (UniqueName: \"kubernetes.io/projected/25311b12-0544-48e2-8b00-f56aff90596d-kube-api-access-sfd9g\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855198    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-etc-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855239    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-env-overrides\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855246    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-ovnkube-config\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855249    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/425443c8-69c7-464e-9f85-0578ff4effdc-config-volume\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855297    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-default-certificate\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855328    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-registration-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855372    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-host\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855397    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-slash\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855429    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-env-overrides\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855429    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-config\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855482    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-hostroot\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.855522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855525    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-multus-certs\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855545    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-default-certificate\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855548    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-trusted-ca-bundle\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855560    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/425443c8-69c7-464e-9f85-0578ff4effdc-tmp-dir\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855594    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-kubelet\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855610    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/425443c8-69c7-464e-9f85-0578ff4effdc-tmp-dir\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855615    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-registry-certificates\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855623    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3897285-e4e6-45da-bf32-fcadd1f9b506-config\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855674    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlfc\" (UniqueName: \"kubernetes.io/projected/b1e328bc-df61-43b5-a5cf-12a2389903a6-kube-api-access-wwlfc\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855694    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-tmp\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855711    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-tmp\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855734    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-sys-fs\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855762    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-registry-certificates\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855784    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-tmp\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855803    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f59bf5f-da15-4174-a5be-b843478f0191-tmp\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855813    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855838    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-netns\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855867    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855917    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38ce8722-6108-4502-b83d-0a29a46f88b0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855953    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3897285-e4e6-45da-bf32-fcadd1f9b506-serving-cert\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.855981    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-etc-kubernetes\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.856008    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-lib-modules\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.856034    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-kubelet-config\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.856074    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-dbus\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.856077    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/38ce8722-6108-4502-b83d-0a29a46f88b0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:22.856279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.856146    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3897285-e4e6-45da-bf32-fcadd1f9b506-serving-cert\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:22.907357 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.907323    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls8vr\" (UniqueName: \"kubernetes.io/projected/73377bc4-2624-4ed6-a5aa-30228b89cbfb-kube-api-access-ls8vr\") pod \"kas-connection-checker-685895c7c9-wq4hz\" (UID: \"73377bc4-2624-4ed6-a5aa-30228b89cbfb\") " pod="kube-system/kas-connection-checker-685895c7c9-wq4hz"
Oct 31 19:16:22.907485 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.907421    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp6kc\" (UniqueName: \"kubernetes.io/projected/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-kube-api-access-xp6kc\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.907485 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.907420    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dh8\" (UniqueName: \"kubernetes.io/projected/78ed8e27-2cd2-43e5-8ee1-23042619743e-kube-api-access-v9dh8\") pod \"kas-connection-checker-685895c7c9-mjspt\" (UID: \"78ed8e27-2cd2-43e5-8ee1-23042619743e\") " pod="kube-system/kas-connection-checker-685895c7c9-mjspt"
Oct 31 19:16:22.912827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.912808    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-bound-sa-token\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.912956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.912936    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xd5w\" (UniqueName: \"kubernetes.io/projected/91815a3a-773d-42b7-8777-d52bb36e0c8e-kube-api-access-8xd5w\") pod \"migrator-66df5d496b-vzd5v\" (UID: \"91815a3a-773d-42b7-8777-d52bb36e0c8e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v"
Oct 31 19:16:22.913052 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.913038    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knvdl\" (UniqueName: \"kubernetes.io/projected/1b0ae648-a85e-4a86-98f1-db2e9600c67a-kube-api-access-knvdl\") pod \"periodic-gathering-wl28r-zphd5\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") " pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:16:22.913681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.913666    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79fc\" (UniqueName: \"kubernetes.io/projected/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-kube-api-access-v79fc\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.914262 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.914225    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km84r\" (UniqueName: \"kubernetes.io/projected/613da58f-accf-4a50-a1b3-b7dccc52fae7-kube-api-access-km84r\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.914336 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.914305    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqzf2\" (UniqueName: \"kubernetes.io/projected/b38ebb5a-bd63-4e8c-94df-c0a904503c4c-kube-api-access-pqzf2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lfx82\" (UID: \"b38ebb5a-bd63-4e8c-94df-c0a904503c4c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82"
Oct 31 19:16:22.914464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.914440    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prj9t\" (UniqueName: \"kubernetes.io/projected/c1b27776-d610-4efa-9f8e-e08f0161a07f-kube-api-access-prj9t\") pod \"service-ca-operator-544d749c8c-w9r2l\" (UID: \"c1b27776-d610-4efa-9f8e-e08f0161a07f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l"
Oct 31 19:16:22.914987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.914839    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8p8\" (UniqueName: \"kubernetes.io/projected/ac149f85-b47b-4fb7-8ad7-3f985129f292-kube-api-access-7k8p8\") pod \"network-check-source-6f78d59f99-zkvsg\" (UID: \"ac149f85-b47b-4fb7-8ad7-3f985129f292\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zkvsg"
Oct 31 19:16:22.915231 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.915210    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7wg7\" (UniqueName: \"kubernetes.io/projected/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-kube-api-access-g7wg7\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:16:22.915317 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.915262    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdqq\" (UniqueName: \"kubernetes.io/projected/291cf47e-f759-4489-abb5-1bafffff7b25-kube-api-access-nqdqq\") pod \"kas-connection-checker-685895c7c9-vmhw6\" (UID: \"291cf47e-f759-4489-abb5-1bafffff7b25\") " pod="kube-system/kas-connection-checker-685895c7c9-vmhw6"
Oct 31 19:16:22.915538 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.915502    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmw8\" (UniqueName: \"kubernetes.io/projected/a3897285-e4e6-45da-bf32-fcadd1f9b506-kube-api-access-qjmw8\") pod \"console-operator-7cd56b8dbc-nztjc\" (UID: \"a3897285-e4e6-45da-bf32-fcadd1f9b506\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:22.915822 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.915791    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-bound-sa-token\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.915943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.915836    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6vs\" (UniqueName: \"kubernetes.io/projected/0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd-kube-api-access-6d6vs\") pod \"service-ca-5fffc97bdf-hhtc4\" (UID: \"0fad5f6e-d422-4bc0-87c5-fa1b5be1e2cd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hhtc4"
Oct 31 19:16:22.915943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.915849    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvjg\" (UniqueName: \"kubernetes.io/projected/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-kube-api-access-njvjg\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:16:22.915943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.915870    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnkn\" (UniqueName: \"kubernetes.io/projected/50011131-90de-4df2-ba1e-28b2a919dbf0-kube-api-access-gtnkn\") pod \"ingress-canary-62bwf\" (UID: \"50011131-90de-4df2-ba1e-28b2a919dbf0\") " pod="openshift-ingress-canary/ingress-canary-62bwf"
Oct 31 19:16:22.916158 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.916125    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6x56\" (UniqueName: \"kubernetes.io/projected/1297e6ef-5035-497c-9777-c6b4448bceba-kube-api-access-f6x56\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:22.916342 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.916324    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwv2\" (UniqueName: \"kubernetes.io/projected/4e36ee6a-f51a-4182-8bfb-68da77946954-kube-api-access-wcwv2\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.916804 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.916781    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlfc\" (UniqueName: \"kubernetes.io/projected/b1e328bc-df61-43b5-a5cf-12a2389903a6-kube-api-access-wwlfc\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:22.917016 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.916996    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wfph\" (UniqueName: \"kubernetes.io/projected/c70d21f3-352d-45fc-853c-a74f690d1898-kube-api-access-5wfph\") pod \"network-check-target-g5rjx\" (UID: \"c70d21f3-352d-45fc-853c-a74f690d1898\") " pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:22.917098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.917042    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9kt\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-kube-api-access-4b9kt\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.917171 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.917109    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7f9m\" (UniqueName: \"kubernetes.io/projected/425443c8-69c7-464e-9f85-0578ff4effdc-kube-api-access-w7f9m\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:22.917171 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.917123    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-792p5\" (UniqueName: \"kubernetes.io/projected/1f59bf5f-da15-4174-a5be-b843478f0191-kube-api-access-792p5\") pod \"insights-operator-7c4986d449-gdrr2\" (UID: \"1f59bf5f-da15-4174-a5be-b843478f0191\") " pod="openshift-insights/insights-operator-7c4986d449-gdrr2"
Oct 31 19:16:22.917993 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.917623    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b92pj\" (UniqueName: \"kubernetes.io/projected/0ec856c2-103c-45fc-af43-bea87251934d-kube-api-access-b92pj\") pod \"migrator-66df5d496b-ldkvc\" (UID: \"0ec856c2-103c-45fc-af43-bea87251934d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc"
Oct 31 19:16:22.917993 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.917696    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdj6v\" (UniqueName: \"kubernetes.io/projected/9fe381a8-6a81-4e21-96c6-715ecd22ff45-kube-api-access-fdj6v\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:16:22.918105 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.917993    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdfnn\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-kube-api-access-fdfnn\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.918464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.918441    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfd9g\" (UniqueName: \"kubernetes.io/projected/25311b12-0544-48e2-8b00-f56aff90596d-kube-api-access-sfd9g\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:22.918587 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.918480    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5q49\" (UniqueName: \"kubernetes.io/projected/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-kube-api-access-s5q49\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.918587 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.918494    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7sm\" (UniqueName: \"kubernetes.io/projected/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-api-access-8v7sm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.919041 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.919010    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8qfj\" (UniqueName: \"kubernetes.io/projected/c0eaee38-fd4e-489a-a854-39245cd595dc-kube-api-access-p8qfj\") pod \"volume-data-source-validator-58c9f5f46d-24dst\" (UID: \"c0eaee38-fd4e-489a-a854-39245cd595dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-24dst"
Oct 31 19:16:22.921175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.921152    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tlwl\" (UniqueName: \"kubernetes.io/projected/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kube-api-access-4tlwl\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.956564 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956527    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-socket-dir-parent\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.956754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956569    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-systemd\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.956754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956597    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.956754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956621    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.956754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956670    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.956754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956683    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-socket-dir-parent\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.956754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956693    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysconfig\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.956754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956717    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-crio-socket\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.956754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956721    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.956754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956741    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fe381a8-6a81-4e21-96c6-715ecd22ff45-hosts-file\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:16:22.956754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956752    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956722    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956757    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysconfig\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956777    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-crio-socket\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956792    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fe381a8-6a81-4e21-96c6-715ecd22ff45-hosts-file\") pod \"node-resolver-8gxf9\" (UID: \"9fe381a8-6a81-4e21-96c6-715ecd22ff45\") " pod="openshift-dns/node-resolver-8gxf9"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956773    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956832    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-etc-selinux\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956711    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-systemd\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956862    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-bin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956887    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956901    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-etc-selinux\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956917    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-conf\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956939    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-bin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956942    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-log-socket\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956960    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.956967    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-netd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957001    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-log-socket\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957007    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-netd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957026    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957044    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-conf\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957060    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957085    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-multus\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957111    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-k8s-cni-cncf-io\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957125    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-cni-multus\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957129    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.957148    5006 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957170    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-k8s-cni-cncf-io\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.957187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957196    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-conf-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957231    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-multus-conf-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957253    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-cnibin\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957277    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-systemd-units\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957296    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-cnibin\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957170    5006 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957314    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-systemd-units\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.957307    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:23.457278061 +0000 UTC m=+1.986812107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : secret "router-metrics-certs-default" not found
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957360    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-system-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957382    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957409    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-socket-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957423    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-system-cni-dir\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957441    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957469    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-bin\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957480    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-socket-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957513    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-sysctl-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957519    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957558    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-etc-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957560    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-cni-bin\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957596    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-registration-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957600    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-etc-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.957633    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:23.457615804 +0000 UTC m=+1.987149863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : configmap references non-existent config key: service-ca.crt
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.957678    5006 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957690    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-registration-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957695    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-host\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957720    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-slash\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957742    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-hostroot\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957757    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-multus-certs\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.957776    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:23.457758304 +0000 UTC m=+1.987292355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : configmap "kube-rbac-proxy" not found
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957785    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-slash\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957790    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-multus-certs\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957827    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40-host\") pod \"node-ca-t5726\" (UID: \"dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40\") " pod="openshift-image-registry/node-ca-t5726"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957834    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-hostroot\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957840    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-kubelet\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957885    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-kubelet\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957906    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-sys-fs\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957934    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957958    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-netns\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957981    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-sys-fs\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.957985    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958026    5006 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958051    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-etc-kubernetes\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958071    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls podName:25311b12-0544-48e2-8b00-f56aff90596d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:23.458057916 +0000 UTC m=+1.987591960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4d9vb" (UID: "25311b12-0544-48e2-8b00-f56aff90596d") : secret "samples-operator-tls" not found
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958085    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-etc-kubernetes\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958027    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-run-netns\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958101    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-lib-modules\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958130    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-kubelet-config\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958152    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-dbus\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958184    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-ovn\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958221    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-lib-modules\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958228    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-kubelet-config\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958232    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-ovn\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958243    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c39943b-0a43-4184-b616-dda8d9f8545e-dbus\") pod \"global-pull-secret-syncer-vs8zn\" (UID: \"0c39943b-0a43-4184-b616-dda8d9f8545e\") " pod="kube-system/global-pull-secret-syncer-vs8zn"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958263    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-host\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958289    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:22.958274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958308    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-node-log\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958331    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-device-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958343    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-host\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958356    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958380    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-run\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958390    5006 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958398    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b37ff40c-3db6-4467-aeaa-c23407b5fce5-device-dir\") pod \"aws-ebs-csi-driver-node-wx44l\" (UID: \"b37ff40c-3db6-4467-aeaa-c23407b5fce5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wx44l"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958401    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-sys\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958437    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-netns\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958444    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert podName:38ce8722-6108-4502-b83d-0a29a46f88b0 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:23.458429634 +0000 UTC m=+1.987963684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jcp2j" (UID: "38ce8722-6108-4502-b83d-0a29a46f88b0") : secret "networking-console-plugin-cert" not found
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958388    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-node-log\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958446    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-sys\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958464    5006 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958477    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-run-netns\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958486    5006 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c7865b76-fzmxv: secret "image-registry-tls" not found
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958471    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958517    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958464    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-run\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958530    5006 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958556    5006 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f799dcd5-gmv4v: secret "image-registry-tls" not found
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958575    5006 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958536    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls podName:ae953c47-79ec-4415-9f33-d4fbb364d6c6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:23.458523755 +0000 UTC m=+1.988057809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls") pod "image-registry-55c7865b76-fzmxv" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6") : secret "image-registry-tls" not found
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958616    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls podName:74f1221c-9c54-4098-a301-ece68db990eb nodeName:}" failed. No retries permitted until 2026-10-31 19:16:23.458601217 +0000 UTC m=+1.988135262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls") pod "image-registry-65f799dcd5-gmv4v" (UID: "74f1221c-9c54-4098-a301-ece68db990eb") : secret "image-registry-tls" not found
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:22.958643    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:23.458633249 +0000 UTC m=+1.988167316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958692    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-cnibin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958720    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-kubernetes\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958748    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-systemd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958772    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-var-lib-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958781    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-cnibin\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958797    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-host-slash\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958802    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-kubernetes\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958812    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-run-systemd\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958838    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-var-lib-openvswitch\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958850    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4-host-slash\") pod \"iptables-alerter-vg9dn\" (UID: \"a1bf62e9-8bb7-4a02-96ad-de1985ddb7a4\") " pod="openshift-network-operator/iptables-alerter-vg9dn"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958857    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958873    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-system-cni-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958894    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-modprobe-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958917    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-var-lib-kubelet\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958933    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e36ee6a-f51a-4182-8bfb-68da77946954-system-cni-dir\") pod \"multus-additional-cni-plugins-gbmm4\" (UID: \"4e36ee6a-f51a-4182-8bfb-68da77946954\") " pod="openshift-multus/multus-additional-cni-plugins-gbmm4"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958940    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.958972    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-kubelet\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.959020    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-etc-modprobe-d\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.959044    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/613da58f-accf-4a50-a1b3-b7dccc52fae7-host-var-lib-kubelet\") pod \"multus-4njcn\" (UID: \"613da58f-accf-4a50-a1b3-b7dccc52fae7\") " pod="openshift-multus/multus-4njcn"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.959043    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42d6c1fe-fc02-4e60-9d54-3b66eb513ed7-var-lib-kubelet\") pod \"tuned-5psz6\" (UID: \"42d6c1fe-fc02-4e60-9d54-3b66eb513ed7\") " pod="openshift-cluster-node-tuning-operator/tuned-5psz6"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.959108    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c72eeb1-17c0-4dd4-99c3-2509ba36b668-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrnqv\" (UID: \"1c72eeb1-17c0-4dd4-99c3-2509ba36b668\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:22.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.959943    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/425443c8-69c7-464e-9f85-0578ff4effdc-metrics-tls\") pod \"dns-default-tcmmx\" (UID: \"425443c8-69c7-464e-9f85-0578ff4effdc\") " pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:22.961286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.960394    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1e328bc-df61-43b5-a5cf-12a2389903a6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9pglj\" (UID: \"b1e328bc-df61-43b5-a5cf-12a2389903a6\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:22.961286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.960931    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1297e6ef-5035-497c-9777-c6b4448bceba-metrics-certs\") pod \"network-metrics-daemon-nptg8\" (UID: \"1297e6ef-5035-497c-9777-c6b4448bceba\") " pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:22.990837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.990806    5006 scope.go:122] "RemoveContainer" containerID="f9ab0191185107cec3cb6699586632b12dd63fd0ad52afbc12b5b9953e787211"
Oct 31 19:16:22.992979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:22.992957    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"
Oct 31 19:16:23.002074 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.002048    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nptg8"
Oct 31 19:16:23.004286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.004227    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:23.005315 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.005301    5006 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:16:23.045059 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.045020    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:23.271382 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.271352    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:23.401627 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.401528    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj"]
Oct 31 19:16:23.402464 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:23.402422    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e328bc_df61_43b5_a5cf_12a2389903a6.slice/crio-c4c14d34e58d08abef5f78bc226473b5e02c3896626373743debf107f018dfb3 WatchSource:0}: Error finding container c4c14d34e58d08abef5f78bc226473b5e02c3896626373743debf107f018dfb3: Status 404 returned error can't find the container with id c4c14d34e58d08abef5f78bc226473b5e02c3896626373743debf107f018dfb3
Oct 31 19:16:23.465233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.465196    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:23.465233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.465237    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:23.465533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.465351    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:23.465533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.465380    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:23.465533 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465486    5006 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 19:16:23.465709 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465546    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert podName:38ce8722-6108-4502-b83d-0a29a46f88b0 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:24.46553318 +0000 UTC m=+2.995067223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jcp2j" (UID: "38ce8722-6108-4502-b83d-0a29a46f88b0") : secret "networking-console-plugin-cert" not found
Oct 31 19:16:23.465709 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465486    5006 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 19:16:23.465709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.465570    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:23.465709 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465582    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls podName:25311b12-0544-48e2-8b00-f56aff90596d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:24.465573771 +0000 UTC m=+2.995107833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4d9vb" (UID: "25311b12-0544-48e2-8b00-f56aff90596d") : secret "samples-operator-tls" not found
Oct 31 19:16:23.465709 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465618    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:24.465602576 +0000 UTC m=+2.995136611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : configmap references non-existent config key: service-ca.crt
Oct 31 19:16:23.465709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.465672    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:23.465709 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465684    5006 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:23.465709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.465699    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465703    5006 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c7865b76-fzmxv: secret "image-registry-tls" not found
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.465765    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465777    5006 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465756    5006 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465815    5006 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f799dcd5-gmv4v: secret "image-registry-tls" not found
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465806    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls podName:ae953c47-79ec-4415-9f33-d4fbb364d6c6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:24.46579214 +0000 UTC m=+2.995326178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls") pod "image-registry-55c7865b76-fzmxv" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6") : secret "image-registry-tls" not found
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465834    5006 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465882    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:24.465873422 +0000 UTC m=+2.995407457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465893    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls podName:74f1221c-9c54-4098-a301-ece68db990eb nodeName:}" failed. No retries permitted until 2026-10-31 19:16:24.465888387 +0000 UTC m=+2.995422423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls") pod "image-registry-65f799dcd5-gmv4v" (UID: "74f1221c-9c54-4098-a301-ece68db990eb") : secret "image-registry-tls" not found
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.465906    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:24.465898571 +0000 UTC m=+2.995432615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : secret "router-metrics-certs-default" not found
Oct 31 19:16:23.466059 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.465916    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:23.506235 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.506209    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nptg8"]
Oct 31 19:16:23.506463 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:23.506443    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1297e6ef_5035_497c_9777_c6b4448bceba.slice/crio-37d2399bdd2f9da8369734908d2fb0106e24682833187ca807d6308b9b6c8268 WatchSource:0}: Error finding container 37d2399bdd2f9da8369734908d2fb0106e24682833187ca807d6308b9b6c8268: Status 404 returned error can't find the container with id 37d2399bdd2f9da8369734908d2fb0106e24682833187ca807d6308b9b6c8268
Oct 31 19:16:23.508097 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:23.508056    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425443c8_69c7_464e_9f85_0578ff4effdc.slice/crio-be76a37785ef42307be170e1d32fdd218f63162ae9d734b7e1b0b970fe5719b0 WatchSource:0}: Error finding container be76a37785ef42307be170e1d32fdd218f63162ae9d734b7e1b0b970fe5719b0: Status 404 returned error can't find the container with id be76a37785ef42307be170e1d32fdd218f63162ae9d734b7e1b0b970fe5719b0
Oct 31 19:16:23.508527 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.508505    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tcmmx"]
Oct 31 19:16:23.574148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.574109    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:23.673083 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.672992    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nptg8" event={"ID":"1297e6ef-5035-497c-9777-c6b4448bceba","Type":"ContainerStarted","Data":"37d2399bdd2f9da8369734908d2fb0106e24682833187ca807d6308b9b6c8268"}
Oct 31 19:16:23.674382 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.674347    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tcmmx" event={"ID":"425443c8-69c7-464e-9f85-0578ff4effdc","Type":"ContainerStarted","Data":"be76a37785ef42307be170e1d32fdd218f63162ae9d734b7e1b0b970fe5719b0"}
Oct 31 19:16:23.675606 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.675577    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj" event={"ID":"b1e328bc-df61-43b5-a5cf-12a2389903a6","Type":"ContainerStarted","Data":"c4c14d34e58d08abef5f78bc226473b5e02c3896626373743debf107f018dfb3"}
Oct 31 19:16:23.677255 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.677236    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:16:23.677695 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.677682    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/1.log"
Oct 31 19:16:23.677761 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.677711    5006 generic.go:356] "Generic (PLEG): container finished" podID="a3897285-e4e6-45da-bf32-fcadd1f9b506" containerID="6ed983b6930541440e10cf0bea022bc6efa41a3af4b14fceac48651af01c4e50" exitCode=255
Oct 31 19:16:23.677817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.677793    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" event={"ID":"a3897285-e4e6-45da-bf32-fcadd1f9b506","Type":"ContainerDied","Data":"6ed983b6930541440e10cf0bea022bc6efa41a3af4b14fceac48651af01c4e50"}
Oct 31 19:16:23.677876 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.677863    5006 scope.go:122] "RemoveContainer" containerID="f9ab0191185107cec3cb6699586632b12dd63fd0ad52afbc12b5b9953e787211"
Oct 31 19:16:23.678441 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.678420    5006 scope.go:122] "RemoveContainer" containerID="6ed983b6930541440e10cf0bea022bc6efa41a3af4b14fceac48651af01c4e50"
Oct 31 19:16:23.678583 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:23.678568    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nztjc_openshift-console-operator(a3897285-e4e6-45da-bf32-fcadd1f9b506)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" podUID="a3897285-e4e6-45da-bf32-fcadd1f9b506"
Oct 31 19:16:23.825167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.825138    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:23.844380 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:23.844349    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:24.055456 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.055418    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:24.263625 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.263498    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:24.305445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.305410    5006 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:24.476764 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.476728    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:24.476956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.476785    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:24.476956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.476813    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:24.476956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.476841    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:24.476956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.476863    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:24.476956 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.476901    5006 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 19:16:24.476956 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.476955    5006 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.476958    5006 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.476976    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.476954691 +0000 UTC m=+5.006488729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : secret "router-metrics-certs-default" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.476982    5006 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.477004    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.476984437 +0000 UTC m=+5.006518491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : configmap references non-existent config key: service-ca.crt
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.477035    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.477044    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert podName:38ce8722-6108-4502-b83d-0a29a46f88b0 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.47702711 +0000 UTC m=+5.006561161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jcp2j" (UID: "38ce8722-6108-4502-b83d-0a29a46f88b0") : secret "networking-console-plugin-cert" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.476967    5006 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55c7865b76-fzmxv: secret "image-registry-tls" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.477067    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls podName:25311b12-0544-48e2-8b00-f56aff90596d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.477056192 +0000 UTC m=+5.006590268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4d9vb" (UID: "25311b12-0544-48e2-8b00-f56aff90596d") : secret "samples-operator-tls" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.477095    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls podName:ae953c47-79ec-4415-9f33-d4fbb364d6c6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.477080597 +0000 UTC m=+5.006614636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls") pod "image-registry-55c7865b76-fzmxv" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6") : secret "image-registry-tls" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.477094    5006 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.477116    5006 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f799dcd5-gmv4v: secret "image-registry-tls" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.477116    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.477149    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls podName:74f1221c-9c54-4098-a301-ece68db990eb nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.477138737 +0000 UTC m=+5.006672801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls") pod "image-registry-65f799dcd5-gmv4v" (UID: "74f1221c-9c54-4098-a301-ece68db990eb") : secret "image-registry-tls" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.477224    5006 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:24.477267 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.477260    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:26.477248407 +0000 UTC m=+5.006782438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:24.481775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.481744    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:24.553581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.553503    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:24.686847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.686818    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:16:24.687281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:24.687188    5006 scope.go:122] "RemoveContainer" containerID="6ed983b6930541440e10cf0bea022bc6efa41a3af4b14fceac48651af01c4e50"
Oct 31 19:16:24.687392 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:24.687362    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nztjc_openshift-console-operator(a3897285-e4e6-45da-bf32-fcadd1f9b506)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" podUID="a3897285-e4e6-45da-bf32-fcadd1f9b506"
Oct 31 19:16:25.113491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:25.113450    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:25.362704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:25.362618    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:25.414098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:25.414028    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gxf9_9fe381a8-6a81-4e21-96c6-715ecd22ff45/dns-node-resolver/0.log"
Oct 31 19:16:25.695947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:25.695917    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vzd5v" event={"ID":"91815a3a-773d-42b7-8777-d52bb36e0c8e","Type":"ContainerStarted","Data":"02b21576a67631ac5f6135a0a87dd133916726bfffea65666223340fb7024cec"}
Oct 31 19:16:25.698715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:25.698679    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ldkvc" event={"ID":"0ec856c2-103c-45fc-af43-bea87251934d","Type":"ContainerStarted","Data":"e867d6072d4fb2cfa8dd55b8b5b8cfe56b369bd1f64a68a1af75da714398c333"}
Oct 31 19:16:25.771905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:25.771861    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:25.990030 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:25.989905    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:26.204517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.204467    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:26.498297 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.498236    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:26.499400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.499380    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:26.499494 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.499423    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:26.499494 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.499459    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:26.499494 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.499489    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:26.499740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.499516    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:26.499740 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:26.499525    5006 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 19:16:26.499740 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:26.499581    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls podName:25311b12-0544-48e2-8b00-f56aff90596d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:30.499565011 +0000 UTC m=+9.029099053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4d9vb" (UID: "25311b12-0544-48e2-8b00-f56aff90596d") : secret "samples-operator-tls" not found
Oct 31 19:16:26.499740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.499619    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:26.499740 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:26.499625    5006 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:26.499740 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:26.499691    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:30.499677734 +0000 UTC m=+9.029211779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:26.500016 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.499742    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:26.500016 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:26.499870    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:30.49985747 +0000 UTC m=+9.029391505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : configmap references non-existent config key: service-ca.crt
Oct 31 19:16:26.500016 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:26.499940    5006 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 19:16:26.500016 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:26.499974    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs podName:f9c1e13d-3f76-4884-af1a-956dc34fd1d6 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:30.499963101 +0000 UTC m=+9.029497133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs") pod "router-default-ddc6dfd67-fk8lv" (UID: "f9c1e13d-3f76-4884-af1a-956dc34fd1d6") : secret "router-metrics-certs-default" not found
Oct 31 19:16:26.502830 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.502780    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"image-registry-65f799dcd5-gmv4v\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") " pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:26.502935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.502846    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"image-registry-55c7865b76-fzmxv\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") " pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:26.504258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.504235    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38ce8722-6108-4502-b83d-0a29a46f88b0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jcp2j\" (UID: \"38ce8722-6108-4502-b83d-0a29a46f88b0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:26.597607 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.597568    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:26.597773 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.597635    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"
Oct 31 19:16:26.603862 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.603840    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:26.704088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.704061    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:26.707871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.707735    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nptg8" event={"ID":"1297e6ef-5035-497c-9777-c6b4448bceba","Type":"ContainerStarted","Data":"b6004352c197714443054d3a5053b7993909974497609ed7f25bffb717fc940e"}
Oct 31 19:16:26.707871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.707827    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nptg8" event={"ID":"1297e6ef-5035-497c-9777-c6b4448bceba","Type":"ContainerStarted","Data":"d7d221e4de58709b613b942e934bca82f61e5e989960892768533ea66b093a03"}
Oct 31 19:16:26.709726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.709419    5006 generic.go:356] "Generic (PLEG): container finished" podID="1f59bf5f-da15-4174-a5be-b843478f0191" containerID="cb6021c50dd5a4a89277ea557aea8b80b4419b129f8e7e1089b9389447f68d22" exitCode=0
Oct 31 19:16:26.709726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.709508    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gdrr2" event={"ID":"1f59bf5f-da15-4174-a5be-b843478f0191","Type":"ContainerDied","Data":"cb6021c50dd5a4a89277ea557aea8b80b4419b129f8e7e1089b9389447f68d22"}
Oct 31 19:16:26.710667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.710318    5006 scope.go:122] "RemoveContainer" containerID="cb6021c50dd5a4a89277ea557aea8b80b4419b129f8e7e1089b9389447f68d22"
Oct 31 19:16:26.714555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.714034    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tcmmx" event={"ID":"425443c8-69c7-464e-9f85-0578ff4effdc","Type":"ContainerStarted","Data":"79900a70a368e02eb6f955db1b06561b67678fa4aa7077b9acdc5d3e87664d23"}
Oct 31 19:16:26.714555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.714075    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tcmmx" event={"ID":"425443c8-69c7-464e-9f85-0578ff4effdc","Type":"ContainerStarted","Data":"8208c01f1aef9bfd3c4da1ed2f6e4540979d1bd5208a1eb8e032d85628ae13d1"}
Oct 31 19:16:26.714555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.714511    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:26.716872 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.716475    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9pglj" event={"ID":"b1e328bc-df61-43b5-a5cf-12a2389903a6","Type":"ContainerStarted","Data":"68349880feeac88c63a234a36bc5c5fa527271122f391c48eaaa88442c09c635"}
Oct 31 19:16:26.814547 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:26.814520    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t5726_dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40/node-ca/0.log"
Oct 31 19:16:27.028762 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.028731    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jcp2j"]
Oct 31 19:16:27.030212 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:27.030181    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ce8722_6108_4502_b83d_0a29a46f88b0.slice/crio-da9d425be03c93720f2d31ec4866b01ad9e10bd7cb295ce13c551d3afc6187f4 WatchSource:0}: Error finding container da9d425be03c93720f2d31ec4866b01ad9e10bd7cb295ce13c551d3afc6187f4: Status 404 returned error can't find the container with id da9d425be03c93720f2d31ec4866b01ad9e10bd7cb295ce13c551d3afc6187f4
Oct 31 19:16:27.032698 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.032229    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55c7865b76-fzmxv"]
Oct 31 19:16:27.032698 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:27.032335    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae953c47_79ec_4415_9f33_d4fbb364d6c6.slice/crio-65431021bf7e8d6210cddb4f0390b2b01718059da062827cd469ef1f44dca416 WatchSource:0}: Error finding container 65431021bf7e8d6210cddb4f0390b2b01718059da062827cd469ef1f44dca416: Status 404 returned error can't find the container with id 65431021bf7e8d6210cddb4f0390b2b01718059da062827cd469ef1f44dca416
Oct 31 19:16:27.042722 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.042692    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65f799dcd5-gmv4v"]
Oct 31 19:16:27.072302 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:27.072262    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f1221c_9c54_4098_a301_ece68db990eb.slice/crio-4278006a59fed65164261e75377232bdd62fc82df42c7abe5818bbb6bb20f658 WatchSource:0}: Error finding container 4278006a59fed65164261e75377232bdd62fc82df42c7abe5818bbb6bb20f658: Status 404 returned error can't find the container with id 4278006a59fed65164261e75377232bdd62fc82df42c7abe5818bbb6bb20f658
Oct 31 19:16:27.274576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.274539    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:27.419613 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.419453    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ldkvc_0ec856c2-103c-45fc-af43-bea87251934d/migrator/0.log"
Oct 31 19:16:27.487008 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.486978    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:27.614711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.614679    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vzd5v_91815a3a-773d-42b7-8777-d52bb36e0c8e/migrator/0.log"
Oct 31 19:16:27.701163 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.701074    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:27.726895 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.726849    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j" event={"ID":"38ce8722-6108-4502-b83d-0a29a46f88b0","Type":"ContainerStarted","Data":"da9d425be03c93720f2d31ec4866b01ad9e10bd7cb295ce13c551d3afc6187f4"}
Oct 31 19:16:27.729168 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.729138    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v" event={"ID":"74f1221c-9c54-4098-a301-ece68db990eb","Type":"ContainerStarted","Data":"bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519"}
Oct 31 19:16:27.729297 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.729187    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v" event={"ID":"74f1221c-9c54-4098-a301-ece68db990eb","Type":"ContainerStarted","Data":"4278006a59fed65164261e75377232bdd62fc82df42c7abe5818bbb6bb20f658"}
Oct 31 19:16:27.731329 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.731293    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv" event={"ID":"ae953c47-79ec-4415-9f33-d4fbb364d6c6","Type":"ContainerStarted","Data":"71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa"}
Oct 31 19:16:27.731441 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.731345    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv" event={"ID":"ae953c47-79ec-4415-9f33-d4fbb364d6c6","Type":"ContainerStarted","Data":"65431021bf7e8d6210cddb4f0390b2b01718059da062827cd469ef1f44dca416"}
Oct 31 19:16:27.733835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.733802    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gdrr2" event={"ID":"1f59bf5f-da15-4174-a5be-b843478f0191","Type":"ContainerStarted","Data":"c5e82f9a19e403851ca4e46be519e02f823e8ef58fcb8bbcc44d48a86da2a867"}
Oct 31 19:16:27.818244 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.818203    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-lfx82_b38ebb5a-bd63-4e8c-94df-c0a904503c4c/kube-storage-version-migrator-operator/0.log"
Oct 31 19:16:27.923571 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:27.923528    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:28.137597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:28.137566    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:28.359509 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:28.359467    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:28.579202 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:28.579107    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:28.739734 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:28.739688    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62bwf" event={"ID":"50011131-90de-4df2-ba1e-28b2a919dbf0","Type":"ContainerStarted","Data":"b9d0c400943a438f01af96c747b0dbd9e40264d45f497420d4123ab5c8e995a2"}
Oct 31 19:16:28.739734 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:28.739730    5006 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 19:16:28.740141 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:28.740059    5006 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 19:16:28.802467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:28.802440    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:29.513249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:29.513216    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:29.513554 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:29.513431    5006 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 19:16:29.567268 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:29.567228    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:29.569167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:29.569150    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:29.748673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:29.748624    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jcp2j" event={"ID":"38ce8722-6108-4502-b83d-0a29a46f88b0","Type":"ContainerStarted","Data":"8a41bcd951eaccb8177b69283edcf1dec2d1b9bae3b0d6df1196cf4f763c6eb1"}
Oct 31 19:16:29.781865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:29.781781    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:29.988619 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:29.988586    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:30.203267 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.203236    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:30.424436 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.424402    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:30.538551 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.538459    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:30.538551 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.538524    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:30.538551 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.538553    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:30.538798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.538588    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:30.538798 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:30.538712    5006 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:30.538798 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:30.538774    5006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls podName:ed0af9e6-dea0-442a-b9a5-50adacf2f3e7 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:38.538760987 +0000 UTC m=+17.068295018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnscp" (UID: "ed0af9e6-dea0-442a-b9a5-50adacf2f3e7") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:16:30.539171 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.539142    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-service-ca-bundle\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:30.541020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.540996    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25311b12-0544-48e2-8b00-f56aff90596d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4d9vb\" (UID: \"25311b12-0544-48e2-8b00-f56aff90596d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:30.541020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.541017    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1e13d-3f76-4884-af1a-956dc34fd1d6-metrics-certs\") pod \"router-default-ddc6dfd67-fk8lv\" (UID: \"f9c1e13d-3f76-4884-af1a-956dc34fd1d6\") " pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:30.795731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.795639    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:30.796704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:30.796684    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"
Oct 31 19:16:31.259961 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.259929    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb"]
Oct 31 19:16:31.262386 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.262365    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddc6dfd67-fk8lv"]
Oct 31 19:16:31.304266 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:31.304237    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c1e13d_3f76_4884_af1a_956dc34fd1d6.slice/crio-c153113497cf64b05623cd5661fb505e367e3f78c675f8431e5c4b850fa6a18a WatchSource:0}: Error finding container c153113497cf64b05623cd5661fb505e367e3f78c675f8431e5c4b850fa6a18a: Status 404 returned error can't find the container with id c153113497cf64b05623cd5661fb505e367e3f78c675f8431e5c4b850fa6a18a
Oct 31 19:16:31.568455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.568367    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:31.586355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.586323    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:31.586783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.586769    5006 scope.go:122] "RemoveContainer" containerID="6ed983b6930541440e10cf0bea022bc6efa41a3af4b14fceac48651af01c4e50"
Oct 31 19:16:31.587006 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:16:31.586986    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nztjc_openshift-console-operator(a3897285-e4e6-45da-bf32-fcadd1f9b506)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" podUID="a3897285-e4e6-45da-bf32-fcadd1f9b506"
Oct 31 19:16:31.756714 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.756679    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb" event={"ID":"25311b12-0544-48e2-8b00-f56aff90596d","Type":"ContainerStarted","Data":"cc43ed546b5ce1a36c12e127bdb20c23d96433353f8e90636264bb69496fdad9"}
Oct 31 19:16:31.758216 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.758189    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddc6dfd67-fk8lv" event={"ID":"f9c1e13d-3f76-4884-af1a-956dc34fd1d6","Type":"ContainerStarted","Data":"ec415f6aae50a121dcffdfcfb8fe4c14ad29abdf3bf7b06d0c2a124f686ff9f9"}
Oct 31 19:16:31.758346 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.758230    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddc6dfd67-fk8lv" event={"ID":"f9c1e13d-3f76-4884-af1a-956dc34fd1d6","Type":"ContainerStarted","Data":"c153113497cf64b05623cd5661fb505e367e3f78c675f8431e5c4b850fa6a18a"}
Oct 31 19:16:31.776976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.776948    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:31.796006 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.795977    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:31.798606 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.798583    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:31.983118 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:31.983077    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:32.195413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:32.195377    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:32.270676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:32.270572    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrnqv"
Oct 31 19:16:32.408485 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:32.408442    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:32.762201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:32.762167    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:33.768479 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:33.768448    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb" event={"ID":"25311b12-0544-48e2-8b00-f56aff90596d","Type":"ContainerStarted","Data":"1473994d5b2eaa6b4b63d606c71452bcf0d081f8cd6d35c314b8ddb835e0b911"}
Oct 31 19:16:33.768849 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:33.768500    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4d9vb" event={"ID":"25311b12-0544-48e2-8b00-f56aff90596d","Type":"ContainerStarted","Data":"ad25e72b705d4c059e8b5d27dc5d82f1ccfad63159bff052253e3c86f7979ef6"}
Oct 31 19:16:33.768849 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:33.768550    5006 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 19:16:34.591420 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:34.591384    5006 scope.go:122] "RemoveContainer" containerID="6ed983b6930541440e10cf0bea022bc6efa41a3af4b14fceac48651af01c4e50"
Oct 31 19:16:34.759870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:34.759832    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:34.761255 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:34.761229    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-ddc6dfd67-fk8lv"
Oct 31 19:16:34.774504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:34.774476    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:16:34.774923 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:34.774588    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" event={"ID":"a3897285-e4e6-45da-bf32-fcadd1f9b506","Type":"ContainerStarted","Data":"9474b013994747747e39f04d602965a6e2d5c737199eab9c7c58be61689f4beb"}
Oct 31 19:16:34.775199 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:34.775183    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:34.776299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:34.776275    5006 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-nztjc container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.22:8443/readyz\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body=
Oct 31 19:16:34.776434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:34.776318    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc" podUID="a3897285-e4e6-45da-bf32-fcadd1f9b506" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.22:8443/readyz\": dial tcp 10.132.0.22:8443: connect: connection refused"
Oct 31 19:16:34.846804 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:34.846708    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:35.783129 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:35.783097    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nztjc"
Oct 31 19:16:36.093932 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:36.093882    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:36.314843 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:36.314811    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:36.534801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:36.534719    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:37.149058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.149027    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:37.393523 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.393479    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:37.602317 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.602283    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:37.742781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.742753    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tcmmx"
Oct 31 19:16:37.801056 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.801028    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7qxh2-hzfbl"]
Oct 31 19:16:37.804580 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.804550    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:37.855009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.854929    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-vhvx7"]
Oct 31 19:16:37.858032 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.858010    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4"]
Oct 31 19:16:37.858156 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.858127    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vhvx7"
Oct 31 19:16:37.861040 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.861019    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4"
Oct 31 19:16:37.902438 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.902407    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55c7865b76-fzmxv"]
Oct 31 19:16:37.902844 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.902819    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-service-ca-bundle\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:37.902970 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.902892    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f2c4306-a1c5-49b0-ab93-cc3c56957a34-tmp\") pod \"downloads-6dd789c8b7-vhvx7\" (UID: \"8f2c4306-a1c5-49b0-ab93-cc3c56957a34\") " pod="openshift-console/downloads-6dd789c8b7-vhvx7"
Oct 31 19:16:37.903092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.903077    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/54073420-a941-4d94-b5db-512984d9597b-archives-path\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:37.903150 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.903106    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74nrj\" (UniqueName: \"kubernetes.io/projected/54073420-a941-4d94-b5db-512984d9597b-kube-api-access-74nrj\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:37.903150 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.903137    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/62ce7d89-451a-4b49-800d-b803eba6758c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5qfg4\" (UID: \"62ce7d89-451a-4b49-800d-b803eba6758c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4"
Oct 31 19:16:37.903317 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.903275    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxcgj\" (UniqueName: \"kubernetes.io/projected/8f2c4306-a1c5-49b0-ab93-cc3c56957a34-kube-api-access-dxcgj\") pod \"downloads-6dd789c8b7-vhvx7\" (UID: \"8f2c4306-a1c5-49b0-ab93-cc3c56957a34\") " pod="openshift-console/downloads-6dd789c8b7-vhvx7"
Oct 31 19:16:37.903412 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.903317    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-trusted-ca-bundle\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:37.911799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.911770    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-z45fh\""
Oct 31 19:16:37.935925 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.935891    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7qxh2-hzfbl"]
Oct 31 19:16:37.953138 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.953107    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54956cf9bf-7nw9v"]
Oct 31 19:16:37.956402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.956378    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:37.960717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.960682    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 19:16:37.960717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.960684    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 19:16:37.984833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.984798    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4"]
Oct 31 19:16:37.986165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:37.986135    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vhvx7"]
Oct 31 19:16:38.004197 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004159    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-registry-certificates\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.004197 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004203    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxcgj\" (UniqueName: \"kubernetes.io/projected/8f2c4306-a1c5-49b0-ab93-cc3c56957a34-kube-api-access-dxcgj\") pod \"downloads-6dd789c8b7-vhvx7\" (UID: \"8f2c4306-a1c5-49b0-ab93-cc3c56957a34\") " pod="openshift-console/downloads-6dd789c8b7-vhvx7"
Oct 31 19:16:38.004397 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004223    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-trusted-ca-bundle\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:38.004397 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004317    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-service-ca-bundle\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:38.004397 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004359    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f2c4306-a1c5-49b0-ab93-cc3c56957a34-tmp\") pod \"downloads-6dd789c8b7-vhvx7\" (UID: \"8f2c4306-a1c5-49b0-ab93-cc3c56957a34\") " pod="openshift-console/downloads-6dd789c8b7-vhvx7"
Oct 31 19:16:38.004540 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004425    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-image-registry-private-configuration\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.004540 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004496    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-registry-tls\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.004540 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004526    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-installation-pull-secrets\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.004728 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004574    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-bound-sa-token\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.004728 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004628    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-trusted-ca\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.004728 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004718    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/54073420-a941-4d94-b5db-512984d9597b-archives-path\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:38.004885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004751    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74nrj\" (UniqueName: \"kubernetes.io/projected/54073420-a941-4d94-b5db-512984d9597b-kube-api-access-74nrj\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:38.004885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004783    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7qr\" (UniqueName: \"kubernetes.io/projected/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-kube-api-access-gz7qr\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.004885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004841    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/62ce7d89-451a-4b49-800d-b803eba6758c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5qfg4\" (UID: \"62ce7d89-451a-4b49-800d-b803eba6758c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4"
Oct 31 19:16:38.004885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004844    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f2c4306-a1c5-49b0-ab93-cc3c56957a34-tmp\") pod \"downloads-6dd789c8b7-vhvx7\" (UID: \"8f2c4306-a1c5-49b0-ab93-cc3c56957a34\") " pod="openshift-console/downloads-6dd789c8b7-vhvx7"
Oct 31 19:16:38.005091 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.004897    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-ca-trust-extracted\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.005151 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.005113    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-service-ca-bundle\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:38.005223 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.005201    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/54073420-a941-4d94-b5db-512984d9597b-archives-path\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:38.005317 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.005299    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-trusted-ca-bundle\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:38.011756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.011734    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kzhrs\""
Oct 31 19:16:38.011904 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.011886    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 19:16:38.018395 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.018365    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/62ce7d89-451a-4b49-800d-b803eba6758c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5qfg4\" (UID: \"62ce7d89-451a-4b49-800d-b803eba6758c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4"
Oct 31 19:16:38.034975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.034946    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54956cf9bf-7nw9v"]
Oct 31 19:16:38.064429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.064399    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74nrj\" (UniqueName: \"kubernetes.io/projected/54073420-a941-4d94-b5db-512984d9597b-kube-api-access-74nrj\") pod \"periodic-gathering-7qxh2-hzfbl\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") " pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:38.064429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.064416    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxcgj\" (UniqueName: \"kubernetes.io/projected/8f2c4306-a1c5-49b0-ab93-cc3c56957a34-kube-api-access-dxcgj\") pod \"downloads-6dd789c8b7-vhvx7\" (UID: \"8f2c4306-a1c5-49b0-ab93-cc3c56957a34\") " pod="openshift-console/downloads-6dd789c8b7-vhvx7"
Oct 31 19:16:38.090587 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.090556    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:38.106019 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.105947    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-trusted-ca\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.106019 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.105977    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7qr\" (UniqueName: \"kubernetes.io/projected/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-kube-api-access-gz7qr\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.106019 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.105996    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-ca-trust-extracted\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.106019 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.106017    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-registry-certificates\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.106313 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.106085    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-image-registry-private-configuration\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.106313 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.106141    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-registry-tls\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.106313 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.106192    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-installation-pull-secrets\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.106313 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.106220    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-bound-sa-token\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.106521 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.106463    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-ca-trust-extracted\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.107041 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.107013    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-registry-certificates\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.107334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.107314    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-trusted-ca\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.108843 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.108816    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-installation-pull-secrets\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.108936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.108818    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-image-registry-private-configuration\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.108936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.108878    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-registry-tls\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.115700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.115677    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:16:38.163161 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.163135    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7qr\" (UniqueName: \"kubernetes.io/projected/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-kube-api-access-gz7qr\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.163498 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.163138    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fc64091-1e1b-47ac-9cb9-6329fe6c0405-bound-sa-token\") pod \"image-registry-54956cf9bf-7nw9v\" (UID: \"6fc64091-1e1b-47ac-9cb9-6329fe6c0405\") " pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.173076 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.173049    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vhvx7"
Oct 31 19:16:38.179622 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.179596    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4"
Oct 31 19:16:38.268952 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.268908    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.305778 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.305747    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:38.482058 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:38.482029    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54073420_a941_4d94_b5db_512984d9597b.slice/crio-1e9ca63a7908065322d4b4d02a836362b4fc2c116f036b0ae3ee98b9649ca780 WatchSource:0}: Error finding container 1e9ca63a7908065322d4b4d02a836362b4fc2c116f036b0ae3ee98b9649ca780: Status 404 returned error can't find the container with id 1e9ca63a7908065322d4b4d02a836362b4fc2c116f036b0ae3ee98b9649ca780
Oct 31 19:16:38.483528 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.483505    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7qxh2-hzfbl"]
Oct 31 19:16:38.546882 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:38.546835    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2c4306_a1c5_49b0_ab93_cc3c56957a34.slice/crio-0bd7d949b5e65f198bb831603883a8f7874eee8e97489bb1fe02c1fe67e8a4f3 WatchSource:0}: Error finding container 0bd7d949b5e65f198bb831603883a8f7874eee8e97489bb1fe02c1fe67e8a4f3: Status 404 returned error can't find the container with id 0bd7d949b5e65f198bb831603883a8f7874eee8e97489bb1fe02c1fe67e8a4f3
Oct 31 19:16:38.547931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.547780    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vhvx7"]
Oct 31 19:16:38.571366 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.571332    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:38.583542 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:38.583505    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ce7d89_451a_4b49_800d_b803eba6758c.slice/crio-8bfe2cba0837e63d00bbc46b4d2608b56484da88bba9a50906859a52485b85ef WatchSource:0}: Error finding container 8bfe2cba0837e63d00bbc46b4d2608b56484da88bba9a50906859a52485b85ef: Status 404 returned error can't find the container with id 8bfe2cba0837e63d00bbc46b4d2608b56484da88bba9a50906859a52485b85ef
Oct 31 19:16:38.585044 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.585014    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4"]
Oct 31 19:16:38.610928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.610898    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:38.613249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.613222    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed0af9e6-dea0-442a-b9a5-50adacf2f3e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnscp\" (UID: \"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7\") " pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:38.641764 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:38.641719    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc64091_1e1b_47ac_9cb9_6329fe6c0405.slice/crio-fd69c57bddc50ba89e0e7c83a16505fad9b4f41f2b385687523364b741e3d364 WatchSource:0}: Error finding container fd69c57bddc50ba89e0e7c83a16505fad9b4f41f2b385687523364b741e3d364: Status 404 returned error can't find the container with id fd69c57bddc50ba89e0e7c83a16505fad9b4f41f2b385687523364b741e3d364
Oct 31 19:16:38.649731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.649703    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54956cf9bf-7nw9v"]
Oct 31 19:16:38.778775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.778684    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:38.793452 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.793419    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7qxh2-hzfbl" event={"ID":"54073420-a941-4d94-b5db-512984d9597b","Type":"ContainerStarted","Data":"eccff544a162755c2c22999b39393ff5f38caa5ae3720aa9105aceb3011b20c0"}
Oct 31 19:16:38.793636 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.793472    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7qxh2-hzfbl" event={"ID":"54073420-a941-4d94-b5db-512984d9597b","Type":"ContainerStarted","Data":"1e9ca63a7908065322d4b4d02a836362b4fc2c116f036b0ae3ee98b9649ca780"}
Oct 31 19:16:38.794786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.794758    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vhvx7" event={"ID":"8f2c4306-a1c5-49b0-ab93-cc3c56957a34","Type":"ContainerStarted","Data":"0bd7d949b5e65f198bb831603883a8f7874eee8e97489bb1fe02c1fe67e8a4f3"}
Oct 31 19:16:38.796409 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.796375    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v" event={"ID":"6fc64091-1e1b-47ac-9cb9-6329fe6c0405","Type":"ContainerStarted","Data":"7c8f019640e8abed639b14bf9796fb7774f9968e4bcc3923c8c7f57b353c5d48"}
Oct 31 19:16:38.796503 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.796423    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v" event={"ID":"6fc64091-1e1b-47ac-9cb9-6329fe6c0405","Type":"ContainerStarted","Data":"fd69c57bddc50ba89e0e7c83a16505fad9b4f41f2b385687523364b741e3d364"}
Oct 31 19:16:38.796573 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.796504    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:16:38.797607 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.797582    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4" event={"ID":"62ce7d89-451a-4b49-800d-b803eba6758c","Type":"ContainerStarted","Data":"8bfe2cba0837e63d00bbc46b4d2608b56484da88bba9a50906859a52485b85ef"}
Oct 31 19:16:38.905855 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.905816    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fnscp"
Oct 31 19:16:38.908779 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:38.908715    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7qxh2-hzfbl" podStartSLOduration=1.908696784 podStartE2EDuration="1.908696784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:38.906960965 +0000 UTC m=+17.436495035" watchObservedRunningTime="2026-10-31 19:16:38.908696784 +0000 UTC m=+17.438230838"
Oct 31 19:16:39.038046 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:39.037943    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v" podStartSLOduration=2.03792963 podStartE2EDuration="2.03792963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:39.03712282 +0000 UTC m=+17.566656894" watchObservedRunningTime="2026-10-31 19:16:39.03792963 +0000 UTC m=+17.567463660"
Oct 31 19:16:39.283304 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:39.283273    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0af9e6_dea0_442a_b9a5_50adacf2f3e7.slice/crio-1c40f2a193c64cf9d6ab0b6632f4a0ea347c16ef8a577eb0b7e5c3419f42e7d9 WatchSource:0}: Error finding container 1c40f2a193c64cf9d6ab0b6632f4a0ea347c16ef8a577eb0b7e5c3419f42e7d9: Status 404 returned error can't find the container with id 1c40f2a193c64cf9d6ab0b6632f4a0ea347c16ef8a577eb0b7e5c3419f42e7d9
Oct 31 19:16:39.284470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:39.284448    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fnscp"]
Oct 31 19:16:39.805240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:39.805201    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fnscp" event={"ID":"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7","Type":"ContainerStarted","Data":"9a836d26db64f3ec6d6c70e11c869ef7ff80f3827e6d8e44db4affdd8f8aa85a"}
Oct 31 19:16:39.805443 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:39.805276    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fnscp" event={"ID":"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7","Type":"ContainerStarted","Data":"1c40f2a193c64cf9d6ab0b6632f4a0ea347c16ef8a577eb0b7e5c3419f42e7d9"}
Oct 31 19:16:40.812879 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:40.812833    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4" event={"ID":"62ce7d89-451a-4b49-800d-b803eba6758c","Type":"ContainerStarted","Data":"70360f79923bdab236d21d0f592399249f81f88f57eb73b0b3786e638a0de276"}
Oct 31 19:16:40.813286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:40.812997    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4"
Oct 31 19:16:40.815274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:40.815232    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fnscp" event={"ID":"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7","Type":"ContainerStarted","Data":"227265fae167945231538a7d85066cf2fd04153bb9789b75f2b6952fdbc77322"}
Oct 31 19:16:40.818801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:40.818776    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4"
Oct 31 19:16:40.828505 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:40.828480    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tcmmx_425443c8-69c7-464e-9f85-0578ff4effdc/dns/0.log"
Oct 31 19:16:40.926947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:40.926883    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5qfg4" podStartSLOduration=2.202240584 podStartE2EDuration="3.92686224s" totalImagesPullingTime="1.724621656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:40.92569638 +0000 UTC m=+19.455230433" watchObservedRunningTime="2026-10-31 19:16:40.92686224 +0000 UTC m=+19.456396295"
Oct 31 19:16:41.028619 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:41.028590    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tcmmx_425443c8-69c7-464e-9f85-0578ff4effdc/kube-rbac-proxy/0.log"
Oct 31 19:16:41.628957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:41.628921    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gxf9_9fe381a8-6a81-4e21-96c6-715ecd22ff45/dns-node-resolver/0.log"
Oct 31 19:16:41.991529 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:41.991443    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:42.204548 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:42.204478    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:42.227767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:42.227742    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54956cf9bf-7nw9v_6fc64091-1e1b-47ac-9cb9-6329fe6c0405/registry/0.log"
Oct 31 19:16:42.421575 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:42.421532    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:42.428411 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:42.428383    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55c7865b76-fzmxv_ae953c47-79ec-4415-9f33-d4fbb364d6c6/registry/0.log"
Oct 31 19:16:42.628890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:42.628859    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65f799dcd5-gmv4v_74f1221c-9c54-4098-a301-ece68db990eb/registry/0.log"
Oct 31 19:16:42.629060 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:42.628925    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:43.228322 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:43.228237    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t5726_dccd8e59-1ea3-43ab-bf6e-f2aa341e3a40/node-ca/0.log"
Oct 31 19:16:43.288977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:43.288928    5006 patch_prober.go:29] interesting pod/image-registry-65f799dcd5-gmv4v container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 19:16:43.289147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:43.288992    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v" podUID="74f1221c-9c54-4098-a301-ece68db990eb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:16:43.429136 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:43.429105    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddc6dfd67-fk8lv_f9c1e13d-3f76-4884-af1a-956dc34fd1d6/router/0.log"
Oct 31 19:16:43.832140 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:43.832103    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62bwf_50011131-90de-4df2-ba1e-28b2a919dbf0/serve-healthcheck-canary/0.log"
Oct 31 19:16:43.835015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:43.834980    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fnscp" event={"ID":"ed0af9e6-dea0-442a-b9a5-50adacf2f3e7","Type":"ContainerStarted","Data":"d433a2b3c83d732d3bf77056d120aeb2358ae47f3722585e358e19bcff08ec03"}
Oct 31 19:16:46.683897 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:46.683862    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:46.962029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:46.961951    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kttjn"
Oct 31 19:16:47.423887 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:47.423854    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:47.647088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:47.647052    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:47.910734 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:47.910704    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:16:47.988767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:47.988721    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:48.209399 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:48.209121    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:48.425927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:48.425873    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:48.638703 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:48.638641    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:48.861790 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:48.861758    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:49.025674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.025573    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nllvs"]
Oct 31 19:16:49.045894 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.045851    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.096840 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.096807    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:49.115525 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.115492    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-wtmp\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.115746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.115532    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-accelerators-collector-config\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.115746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.115562    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5sq5\" (UniqueName: \"kubernetes.io/projected/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-kube-api-access-t5sq5\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.115746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.115709    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-sys\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.115928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.115753    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.115928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.115784    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-tls\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.115928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.115821    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-root\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.116084 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.115936    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-textfile\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.116084 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.115971    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-metrics-client-ca\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.169533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.169500    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 19:16:49.214037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.213948    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 19:16:49.214037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.214003    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mvdsq\""
Oct 31 19:16:49.214256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.214045    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 19:16:49.214256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.214152    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 19:16:49.217521 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217492    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-textfile\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.217701 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217533    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-metrics-client-ca\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.217701 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217619    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-wtmp\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.217701 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217642    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-accelerators-collector-config\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.217701 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217680    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5sq5\" (UniqueName: \"kubernetes.io/projected/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-kube-api-access-t5sq5\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.217931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217709    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-sys\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.217931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217729    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.217931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217747    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-tls\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.217931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217764    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-root\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.217931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217814    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-wtmp\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.218153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217938    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-sys\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.218352 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.217856    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-root\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.218631 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.218601    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-textfile\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.219165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.219133    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-accelerators-collector-config\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.219853 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.219831    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-metrics-client-ca\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.221895 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.221876    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.222778 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.222632    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-node-exporter-tls\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.276137 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.276060    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5sq5\" (UniqueName: \"kubernetes.io/projected/20fab3dd-c950-4c2e-b47a-f1dd3c9b1105-kube-api-access-t5sq5\") pod \"node-exporter-nllvs\" (UID: \"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105\") " pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.339663 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.339609    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:49.419001 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.418962    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nllvs"
Oct 31 19:16:49.457247 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:16:49.457200    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fab3dd_c950_4c2e_b47a_f1dd3c9b1105.slice/crio-baf7626ca246eb78e69f146feaadff504be6540126ae3a15593dde46a41f5ff6 WatchSource:0}: Error finding container baf7626ca246eb78e69f146feaadff504be6540126ae3a15593dde46a41f5ff6: Status 404 returned error can't find the container with id baf7626ca246eb78e69f146feaadff504be6540126ae3a15593dde46a41f5ff6
Oct 31 19:16:49.563303 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.563184    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:49.753746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.753717    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:16:49.863809 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:49.863774    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nllvs" event={"ID":"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105","Type":"ContainerStarted","Data":"baf7626ca246eb78e69f146feaadff504be6540126ae3a15593dde46a41f5ff6"}
Oct 31 19:16:50.461183 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:50.461146    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:50.687274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:50.687236    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:50.899116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:50.899075    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:51.341251 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:51.341218    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:51.875641 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:51.875603    5006 generic.go:356] "Generic (PLEG): container finished" podID="20fab3dd-c950-4c2e-b47a-f1dd3c9b1105" containerID="bd0e49096ca339e3fbeef052897d514ad3c917d436fabbb8ae25f9cf91342cf3" exitCode=0
Oct 31 19:16:51.876126 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:51.875690    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nllvs" event={"ID":"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105","Type":"ContainerDied","Data":"bd0e49096ca339e3fbeef052897d514ad3c917d436fabbb8ae25f9cf91342cf3"}
Oct 31 19:16:52.433825 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:52.433784    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:52.662976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:52.662941    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:52.882731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:52.882693    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nllvs" event={"ID":"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105","Type":"ContainerStarted","Data":"ba76b4862fe31519860e69f9680bcec55ffd56e309b4afbcc2dcaf32a0030052"}
Oct 31 19:16:52.883109 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:52.882760    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nllvs" event={"ID":"20fab3dd-c950-4c2e-b47a-f1dd3c9b1105","Type":"ContainerStarted","Data":"877b6bb969b7ed2db5decf2d1f70ff2edce7ec9550444aec44cf0afc436c6947"}
Oct 31 19:16:53.046307 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:53.046238    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nllvs" podStartSLOduration=2.920874129 podStartE2EDuration="5.046217752s" totalImagesPullingTime="2.024568755s" totalInitContainerRuntime="100.774868ms" isStatefulPod=true podCreationTimestamp="2026-10-31 19:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:53.044564142 +0000 UTC m=+31.574098222" watchObservedRunningTime="2026-10-31 19:16:53.046217752 +0000 UTC m=+31.575751836"
Oct 31 19:16:53.076577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:53.076505    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:53.293998 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:53.293923    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:53.508947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:53.508887    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:53.721644 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:53.721607    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:53.935359 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:53.935294    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:54.557069 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:54.557033    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g5rjx"
Oct 31 19:16:55.898720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:55.898674    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vhvx7" event={"ID":"8f2c4306-a1c5-49b0-ab93-cc3c56957a34","Type":"ContainerStarted","Data":"27f7536c0cd02d6244ca200f9d3a3d3e490874231a50f6d3c0ddf863135cfed2"}
Oct 31 19:16:55.899207 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:55.898978    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-vhvx7"
Oct 31 19:16:55.913349 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:55.913316    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-vhvx7"
Oct 31 19:16:56.114066 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:56.114008    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-vhvx7" podStartSLOduration=2.383234009 podStartE2EDuration="19.113990189s" totalImagesPullingTime="16.73075618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:56.062991015 +0000 UTC m=+34.592525097" watchObservedRunningTime="2026-10-31 19:16:56.113990189 +0000 UTC m=+34.643524236"
Oct 31 19:16:57.910929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:57.910863    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:58.132351 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:58.132293    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:58.352244 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:58.352178    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:59.810185 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:16:59.810148    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54956cf9bf-7nw9v"
Oct 31 19:17:00.032777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:00.032738    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65f799dcd5-gmv4v"]
Oct 31 19:17:00.180791 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:00.180751    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:00.392112 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:00.392078    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:00.957130 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:00.957054    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:01.192722 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:01.192500    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:02.923245 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:02.923175    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv" podUID="ae953c47-79ec-4415-9f33-d4fbb364d6c6" containerName="registry" containerID="cri-o://71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa" gracePeriod=30
Oct 31 19:17:03.117288 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.117240    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:03.227221 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.227181    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:17:03.277664 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.277615    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-certificates\") pod \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") "
Oct 31 19:17:03.277853 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.277733    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-installation-pull-secrets\") pod \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") "
Oct 31 19:17:03.277853 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.277776    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-kube-api-access-fdfnn\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-kube-api-access-fdfnn\") pod \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") "
Oct 31 19:17:03.277853 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.277804    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-trusted-ca\") pod \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") "
Oct 31 19:17:03.278037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.277885    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-image-registry-private-configuration\") pod \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") "
Oct 31 19:17:03.278037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.277945    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-bound-sa-token\") pod \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") "
Oct 31 19:17:03.278037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.277981    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae953c47-79ec-4415-9f33-d4fbb364d6c6-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae953c47-79ec-4415-9f33-d4fbb364d6c6-ca-trust-extracted\") pod \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") "
Oct 31 19:17:03.278037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.278021    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") pod \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\" (UID: \"ae953c47-79ec-4415-9f33-d4fbb364d6c6\") "
Oct 31 19:17:03.278227 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.278169    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-certificates" pod "ae953c47-79ec-4415-9f33-d4fbb364d6c6" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:17:03.278287 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.278219    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-trusted-ca" pod "ae953c47-79ec-4415-9f33-d4fbb364d6c6" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:17:03.278352 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.278314    5006 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-trusted-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.278352 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.278335    5006 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-certificates\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.281302 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.281157    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-bound-sa-token" pod "ae953c47-79ec-4415-9f33-d4fbb364d6c6" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:03.281820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.281304    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-image-registry-private-configuration" pod "ae953c47-79ec-4415-9f33-d4fbb364d6c6" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:17:03.281820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.281616    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-installation-pull-secrets" pod "ae953c47-79ec-4415-9f33-d4fbb364d6c6" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:17:03.282219 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.282188    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls" pod "ae953c47-79ec-4415-9f33-d4fbb364d6c6" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:03.282700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.282637    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-kube-api-access-fdfnn" pod "ae953c47-79ec-4415-9f33-d4fbb364d6c6" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6"). InnerVolumeSpecName "kube-api-access-fdfnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:03.287139 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.287114    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae953c47-79ec-4415-9f33-d4fbb364d6c6-ca-trust-extracted" pod "ae953c47-79ec-4415-9f33-d4fbb364d6c6" (UID: "ae953c47-79ec-4415-9f33-d4fbb364d6c6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:03.339985 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.339941    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:03.379350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.379315    5006 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-image-registry-private-configuration\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.379350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.379349    5006 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-bound-sa-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.379592 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.379367    5006 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae953c47-79ec-4415-9f33-d4fbb364d6c6-ca-trust-extracted\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.379592 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.379381    5006 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-registry-tls\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.379592 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.379396    5006 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae953c47-79ec-4415-9f33-d4fbb364d6c6-installation-pull-secrets\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.379592 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.379412    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdfnn\" (UniqueName: \"kubernetes.io/projected/ae953c47-79ec-4415-9f33-d4fbb364d6c6-kube-api-access-fdfnn\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:03.959593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.959498    5006 generic.go:356] "Generic (PLEG): container finished" podID="ae953c47-79ec-4415-9f33-d4fbb364d6c6" containerID="71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa" exitCode=0
Oct 31 19:17:03.959593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.959551    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv"
Oct 31 19:17:03.960157 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.959602    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv" event={"ID":"ae953c47-79ec-4415-9f33-d4fbb364d6c6","Type":"ContainerDied","Data":"71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa"}
Oct 31 19:17:03.960157 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.959690    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55c7865b76-fzmxv" event={"ID":"ae953c47-79ec-4415-9f33-d4fbb364d6c6","Type":"ContainerDied","Data":"65431021bf7e8d6210cddb4f0390b2b01718059da062827cd469ef1f44dca416"}
Oct 31 19:17:03.960157 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.959720    5006 scope.go:122] "RemoveContainer" containerID="71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa"
Oct 31 19:17:03.970622 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.970598    5006 scope.go:122] "RemoveContainer" containerID="71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa"
Oct 31 19:17:03.971011 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:17:03.970985    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa\": container with ID starting with 71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa not found: ID does not exist" containerID="71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa"
Oct 31 19:17:03.971112 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:03.971019    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa"} err="failed to get container status \"71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa\": rpc error: code = NotFound desc = could not find container \"71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa\": container with ID starting with 71c4fa8de16b3a40d1265329e7c89c92117e92be540290ca4140610e739d32fa not found: ID does not exist"
Oct 31 19:17:04.126082 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:04.126048    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55c7865b76-fzmxv"]
Oct 31 19:17:04.176520 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:04.176486    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55c7865b76-fzmxv"]
Oct 31 19:17:04.320104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:04.319981    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:04.595676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:04.595630    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:05.392148 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:17:05.392113    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:17:05.596696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:05.596638    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae953c47-79ec-4415-9f33-d4fbb364d6c6" path="/var/lib/kubelet/pods/ae953c47-79ec-4415-9f33-d4fbb364d6c6/volumes"
Oct 31 19:17:06.181821 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:06.181776    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:06.439816 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:06.439724    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:07.417447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:07.417413    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:07.719947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:07.719875    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:10.279444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.279410    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-wl28r-zphd5"]
Oct 31 19:17:10.279951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.279610    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-wl28r-zphd5" podUID="1b0ae648-a85e-4a86-98f1-db2e9600c67a" containerName="insights-gathering" containerID="cri-o://1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3" gracePeriod=30
Oct 31 19:17:10.581341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.581303    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:17:10.628719 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.628643    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-service-ca-bundle\") pod \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") "
Oct 31 19:17:10.628926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.628808    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b0ae648-a85e-4a86-98f1-db2e9600c67a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1b0ae648-a85e-4a86-98f1-db2e9600c67a-archives-path\") pod \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") "
Oct 31 19:17:10.628926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.628866    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0ae648-a85e-4a86-98f1-db2e9600c67a-kube-api-access-knvdl\" (UniqueName: \"kubernetes.io/projected/1b0ae648-a85e-4a86-98f1-db2e9600c67a-kube-api-access-knvdl\") pod \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") "
Oct 31 19:17:10.628926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.628896    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-trusted-ca-bundle\") pod \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\" (UID: \"1b0ae648-a85e-4a86-98f1-db2e9600c67a\") "
Oct 31 19:17:10.629116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.629085    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-service-ca-bundle" pod "1b0ae648-a85e-4a86-98f1-db2e9600c67a" (UID: "1b0ae648-a85e-4a86-98f1-db2e9600c67a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:17:10.629239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.629157    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0ae648-a85e-4a86-98f1-db2e9600c67a-archives-path" pod "1b0ae648-a85e-4a86-98f1-db2e9600c67a" (UID: "1b0ae648-a85e-4a86-98f1-db2e9600c67a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:10.629346 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.629328    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-trusted-ca-bundle" pod "1b0ae648-a85e-4a86-98f1-db2e9600c67a" (UID: "1b0ae648-a85e-4a86-98f1-db2e9600c67a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:17:10.631310 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.631285    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0ae648-a85e-4a86-98f1-db2e9600c67a-kube-api-access-knvdl" pod "1b0ae648-a85e-4a86-98f1-db2e9600c67a" (UID: "1b0ae648-a85e-4a86-98f1-db2e9600c67a"). InnerVolumeSpecName "kube-api-access-knvdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:10.729733 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.729700    5006 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-service-ca-bundle\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:10.729733 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.729731    5006 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1b0ae648-a85e-4a86-98f1-db2e9600c67a-archives-path\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:10.729927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.729742    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knvdl\" (UniqueName: \"kubernetes.io/projected/1b0ae648-a85e-4a86-98f1-db2e9600c67a-kube-api-access-knvdl\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:10.729927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.729752    5006 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0ae648-a85e-4a86-98f1-db2e9600c67a-trusted-ca-bundle\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:10.993660 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.993615    5006 generic.go:356] "Generic (PLEG): container finished" podID="1b0ae648-a85e-4a86-98f1-db2e9600c67a" containerID="1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3" exitCode=2
Oct 31 19:17:10.993832 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.993690    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wl28r-zphd5"
Oct 31 19:17:10.993832 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.993703    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wl28r-zphd5" event={"ID":"1b0ae648-a85e-4a86-98f1-db2e9600c67a","Type":"ContainerDied","Data":"1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3"}
Oct 31 19:17:10.993832 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.993755    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wl28r-zphd5" event={"ID":"1b0ae648-a85e-4a86-98f1-db2e9600c67a","Type":"ContainerDied","Data":"531aa693e7a74bc14f4db1b4297bb59e2f1f6e8cee45f437058dd7517a9a06e4"}
Oct 31 19:17:10.993832 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:10.993775    5006 scope.go:122] "RemoveContainer" containerID="1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3"
Oct 31 19:17:11.003107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:11.003087    5006 scope.go:122] "RemoveContainer" containerID="1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3"
Oct 31 19:17:11.003424 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:17:11.003401    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3\": container with ID starting with 1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3 not found: ID does not exist" containerID="1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3"
Oct 31 19:17:11.003468 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:11.003436    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3"} err="failed to get container status \"1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3\": rpc error: code = NotFound desc = could not find container \"1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3\": container with ID starting with 1951a435727a2f45b0a4af0d3a094810027356afe6cf7959f0b9775c17fce8f3 not found: ID does not exist"
Oct 31 19:17:11.157407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:11.157373    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-wl28r-zphd5"]
Oct 31 19:17:11.207572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:11.207538    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-wl28r-zphd5"]
Oct 31 19:17:11.595824 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:11.595783    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0ae648-a85e-4a86-98f1-db2e9600c67a" path="/var/lib/kubelet/pods/1b0ae648-a85e-4a86-98f1-db2e9600c67a/volumes"
Oct 31 19:17:11.840482 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:11.840435    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:12.165632 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:12.165580    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:16.641356 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:16.641324    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:16.908515 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:16.908441    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:22.413150 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:22.413116    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:22.642929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:22.642893    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:25.057163 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.057098    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v" podUID="74f1221c-9c54-4098-a301-ece68db990eb" containerName="registry" containerID="cri-o://bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519" gracePeriod=30
Oct 31 19:17:25.356610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.356565    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:17:25.455968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.455930    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-registry-certificates\") pod \"74f1221c-9c54-4098-a301-ece68db990eb\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") "
Oct 31 19:17:25.456165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.455984    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-installation-pull-secrets\") pod \"74f1221c-9c54-4098-a301-ece68db990eb\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") "
Oct 31 19:17:25.456165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.456035    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-kube-api-access-4b9kt\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-kube-api-access-4b9kt\") pod \"74f1221c-9c54-4098-a301-ece68db990eb\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") "
Oct 31 19:17:25.456165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.456075    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f1221c-9c54-4098-a301-ece68db990eb-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74f1221c-9c54-4098-a301-ece68db990eb-ca-trust-extracted\") pod \"74f1221c-9c54-4098-a301-ece68db990eb\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") "
Oct 31 19:17:25.456165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.456093    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-bound-sa-token\") pod \"74f1221c-9c54-4098-a301-ece68db990eb\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") "
Oct 31 19:17:25.456165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.456125    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") pod \"74f1221c-9c54-4098-a301-ece68db990eb\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") "
Oct 31 19:17:25.456165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.456156    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-trusted-ca\") pod \"74f1221c-9c54-4098-a301-ece68db990eb\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") "
Oct 31 19:17:25.456454 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.456178    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-image-registry-private-configuration\") pod \"74f1221c-9c54-4098-a301-ece68db990eb\" (UID: \"74f1221c-9c54-4098-a301-ece68db990eb\") "
Oct 31 19:17:25.456567 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.456512    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-registry-certificates" pod "74f1221c-9c54-4098-a301-ece68db990eb" (UID: "74f1221c-9c54-4098-a301-ece68db990eb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:17:25.456894 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.456857    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-trusted-ca" pod "74f1221c-9c54-4098-a301-ece68db990eb" (UID: "74f1221c-9c54-4098-a301-ece68db990eb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:17:25.458605 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.458548    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-installation-pull-secrets" pod "74f1221c-9c54-4098-a301-ece68db990eb" (UID: "74f1221c-9c54-4098-a301-ece68db990eb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:17:25.458605 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.458553    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls" pod "74f1221c-9c54-4098-a301-ece68db990eb" (UID: "74f1221c-9c54-4098-a301-ece68db990eb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:25.458874 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.458846    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-bound-sa-token" pod "74f1221c-9c54-4098-a301-ece68db990eb" (UID: "74f1221c-9c54-4098-a301-ece68db990eb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:25.459013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.458985    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-kube-api-access-4b9kt" pod "74f1221c-9c54-4098-a301-ece68db990eb" (UID: "74f1221c-9c54-4098-a301-ece68db990eb"). InnerVolumeSpecName "kube-api-access-4b9kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:17:25.459104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.459030    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-image-registry-private-configuration" pod "74f1221c-9c54-4098-a301-ece68db990eb" (UID: "74f1221c-9c54-4098-a301-ece68db990eb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:17:25.463562 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.463537    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f1221c-9c54-4098-a301-ece68db990eb-ca-trust-extracted" pod "74f1221c-9c54-4098-a301-ece68db990eb" (UID: "74f1221c-9c54-4098-a301-ece68db990eb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:17:25.557399 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.557357    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b9kt\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-kube-api-access-4b9kt\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:25.557399 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.557391    5006 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74f1221c-9c54-4098-a301-ece68db990eb-ca-trust-extracted\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:25.557399 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.557403    5006 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-bound-sa-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:25.557682 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.557414    5006 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74f1221c-9c54-4098-a301-ece68db990eb-registry-tls\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:25.557682 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.557423    5006 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-trusted-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:25.557682 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.557434    5006 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-image-registry-private-configuration\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:25.557682 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.557443    5006 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74f1221c-9c54-4098-a301-ece68db990eb-registry-certificates\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:25.557682 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:25.557451    5006 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74f1221c-9c54-4098-a301-ece68db990eb-installation-pull-secrets\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:17:26.057012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:26.056920    5006 generic.go:356] "Generic (PLEG): container finished" podID="74f1221c-9c54-4098-a301-ece68db990eb" containerID="bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519" exitCode=0
Oct 31 19:17:26.057012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:26.056974    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v"
Oct 31 19:17:26.057201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:26.057006    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v" event={"ID":"74f1221c-9c54-4098-a301-ece68db990eb","Type":"ContainerDied","Data":"bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519"}
Oct 31 19:17:26.057201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:26.057063    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f799dcd5-gmv4v" event={"ID":"74f1221c-9c54-4098-a301-ece68db990eb","Type":"ContainerDied","Data":"4278006a59fed65164261e75377232bdd62fc82df42c7abe5818bbb6bb20f658"}
Oct 31 19:17:26.057201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:26.057087    5006 scope.go:122] "RemoveContainer" containerID="bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519"
Oct 31 19:17:26.065576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:26.065553    5006 scope.go:122] "RemoveContainer" containerID="bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519"
Oct 31 19:17:26.065934 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:17:26.065912    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519\": container with ID starting with bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519 not found: ID does not exist" containerID="bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519"
Oct 31 19:17:26.066003 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:26.065944    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519"} err="failed to get container status \"bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519\": rpc error: code = NotFound desc = could not find container \"bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519\": container with ID starting with bdb48de7015152fd877c11c9a93c1b03cba828b86d8e6fde018d5de5c9445519 not found: ID does not exist"
Oct 31 19:17:26.274190 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:26.274139    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65f799dcd5-gmv4v"]
Oct 31 19:17:26.324586 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:26.324236    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65f799dcd5-gmv4v"]
Oct 31 19:17:27.596177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:27.596141    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f1221c-9c54-4098-a301-ece68db990eb" path="/var/lib/kubelet/pods/74f1221c-9c54-4098-a301-ece68db990eb/volumes"
Oct 31 19:17:29.071590 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:29.071556    5006 generic.go:356] "Generic (PLEG): container finished" podID="b38ebb5a-bd63-4e8c-94df-c0a904503c4c" containerID="c00202dc0372a188d238d1ad119f6626093b3efd9977af1495afec190592ce69" exitCode=0
Oct 31 19:17:29.071989 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:29.071625    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82" event={"ID":"b38ebb5a-bd63-4e8c-94df-c0a904503c4c","Type":"ContainerDied","Data":"c00202dc0372a188d238d1ad119f6626093b3efd9977af1495afec190592ce69"}
Oct 31 19:17:29.072034 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:29.071998    5006 scope.go:122] "RemoveContainer" containerID="c00202dc0372a188d238d1ad119f6626093b3efd9977af1495afec190592ce69"
Oct 31 19:17:30.077805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:30.077770    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lfx82" event={"ID":"b38ebb5a-bd63-4e8c-94df-c0a904503c4c","Type":"ContainerStarted","Data":"5eb6b04bd0f9b5d8b3659c160569862969849bd202433662bcee95d2578581e5"}
Oct 31 19:17:35.393010 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:17:35.392976    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:17:37.402798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:37.402758    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:37.614871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:37.614841    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:38.111873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:38.111829    5006 generic.go:356] "Generic (PLEG): container finished" podID="c1b27776-d610-4efa-9f8e-e08f0161a07f" containerID="a81491b9148450764b17496eefc78e77a22ce716b3e7b8391db80237cfed7974" exitCode=0
Oct 31 19:17:38.112044 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:38.111899    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l" event={"ID":"c1b27776-d610-4efa-9f8e-e08f0161a07f","Type":"ContainerDied","Data":"a81491b9148450764b17496eefc78e77a22ce716b3e7b8391db80237cfed7974"}
Oct 31 19:17:38.112249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:38.112234    5006 scope.go:122] "RemoveContainer" containerID="a81491b9148450764b17496eefc78e77a22ce716b3e7b8391db80237cfed7974"
Oct 31 19:17:38.523911 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:38.523831    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:38.790543 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:38.790450    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:39.119111 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:39.119073    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w9r2l" event={"ID":"c1b27776-d610-4efa-9f8e-e08f0161a07f","Type":"ContainerStarted","Data":"850f605d2ba5bd395104596faf25b16cd2d2c6245ec49c5dab47fe83db0397be"}
Oct 31 19:17:50.152957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:50.152906    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:50.404144 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:50.404064    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:50.649406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:50.649370    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:57.246792 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:17:57.246754    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:05.392552 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:18:05.392503    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:18:06.226098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:06.226065    5006 generic.go:356] "Generic (PLEG): container finished" podID="54073420-a941-4d94-b5db-512984d9597b" containerID="eccff544a162755c2c22999b39393ff5f38caa5ae3720aa9105aceb3011b20c0" exitCode=0
Oct 31 19:18:06.226293 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:06.226137    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7qxh2-hzfbl" event={"ID":"54073420-a941-4d94-b5db-512984d9597b","Type":"ContainerDied","Data":"eccff544a162755c2c22999b39393ff5f38caa5ae3720aa9105aceb3011b20c0"}
Oct 31 19:18:07.377174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.377152    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:18:07.411197 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.411167    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54073420-a941-4d94-b5db-512984d9597b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/54073420-a941-4d94-b5db-512984d9597b-archives-path\") pod \"54073420-a941-4d94-b5db-512984d9597b\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") "
Oct 31 19:18:07.411365 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.411215    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54073420-a941-4d94-b5db-512984d9597b-kube-api-access-74nrj\" (UniqueName: \"kubernetes.io/projected/54073420-a941-4d94-b5db-512984d9597b-kube-api-access-74nrj\") pod \"54073420-a941-4d94-b5db-512984d9597b\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") "
Oct 31 19:18:07.411365 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.411238    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-service-ca-bundle\") pod \"54073420-a941-4d94-b5db-512984d9597b\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") "
Oct 31 19:18:07.411365 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.411270    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-trusted-ca-bundle\") pod \"54073420-a941-4d94-b5db-512984d9597b\" (UID: \"54073420-a941-4d94-b5db-512984d9597b\") "
Oct 31 19:18:07.411595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.411568    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54073420-a941-4d94-b5db-512984d9597b-archives-path" pod "54073420-a941-4d94-b5db-512984d9597b" (UID: "54073420-a941-4d94-b5db-512984d9597b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:18:07.411709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.411689    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-trusted-ca-bundle" pod "54073420-a941-4d94-b5db-512984d9597b" (UID: "54073420-a941-4d94-b5db-512984d9597b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:18:07.411755 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.411688    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-service-ca-bundle" pod "54073420-a941-4d94-b5db-512984d9597b" (UID: "54073420-a941-4d94-b5db-512984d9597b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:18:07.413293 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.413271    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54073420-a941-4d94-b5db-512984d9597b-kube-api-access-74nrj" pod "54073420-a941-4d94-b5db-512984d9597b" (UID: "54073420-a941-4d94-b5db-512984d9597b"). InnerVolumeSpecName "kube-api-access-74nrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:18:07.511896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.511814    5006 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/54073420-a941-4d94-b5db-512984d9597b-archives-path\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:18:07.511896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.511845    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74nrj\" (UniqueName: \"kubernetes.io/projected/54073420-a941-4d94-b5db-512984d9597b-kube-api-access-74nrj\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:18:07.511896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.511859    5006 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-service-ca-bundle\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:18:07.511896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:07.511872    5006 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54073420-a941-4d94-b5db-512984d9597b-trusted-ca-bundle\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:18:08.235710 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:08.235644    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7qxh2-hzfbl" event={"ID":"54073420-a941-4d94-b5db-512984d9597b","Type":"ContainerDied","Data":"1e9ca63a7908065322d4b4d02a836362b4fc2c116f036b0ae3ee98b9649ca780"}
Oct 31 19:18:08.235891 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:08.235744    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e9ca63a7908065322d4b4d02a836362b4fc2c116f036b0ae3ee98b9649ca780"
Oct 31 19:18:08.235891 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:08.235659    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7qxh2-hzfbl"
Oct 31 19:18:09.502537 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:09.502504    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:09.728915 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:09.728880    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:12.177441 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:12.177373    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:28.402350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:28.402312    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:28.653555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:28.653467    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:35.392480 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:18:35.392393    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:18:46.477785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:46.477745    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:46.712970 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:46.712930    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:59.028317 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:59.028282    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:59.250452 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:18:59.250413    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:01.391684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:19:01.391627    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:01.762259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:19:01.762159    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:01.998113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:19:01.998080    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:05.392747 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:19:05.392708    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:19:07.406558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:19:07.406522    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:07.660086 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:19:07.660000    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:12.502011 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:19:12.501978    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:35.393198 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:19:35.393157    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:20:05.391056 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:20:05.390977    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:20:11.432863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:20:11.432830    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:13.098142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:20:13.098103    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:35.392994 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:20:35.392949    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:21:05.393148 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:21:05.393110    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:21:13.189936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:13.189898    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:13.611605 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:13.611573    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:13.861563 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:13.861534    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:15.036077 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:15.036041    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:15.263478 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:15.263444    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:15.987129 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:15.987086    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:21.604471 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:21.604441    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:21:21.605241 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:21.605214    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:21:22.085453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.085373    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:22.319485 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.319453    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:22.363935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.363902    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"]
Oct 31 19:21:22.364246 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364231    5006 cpu_manager.go:414] "containerMap: removing container" podUID="54073420-a941-4d94-b5db-512984d9597b" containerName="insights-gathering"
Oct 31 19:21:22.364301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364249    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54073420-a941-4d94-b5db-512984d9597b" containerName="insights-gathering"
Oct 31 19:21:22.364301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364262    5006 cpu_manager.go:414] "containerMap: removing container" podUID="ae953c47-79ec-4415-9f33-d4fbb364d6c6" containerName="registry"
Oct 31 19:21:22.364301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364267    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae953c47-79ec-4415-9f33-d4fbb364d6c6" containerName="registry"
Oct 31 19:21:22.364301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364278    5006 cpu_manager.go:414] "containerMap: removing container" podUID="74f1221c-9c54-4098-a301-ece68db990eb" containerName="registry"
Oct 31 19:21:22.364301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364283    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f1221c-9c54-4098-a301-ece68db990eb" containerName="registry"
Oct 31 19:21:22.364301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364294    5006 cpu_manager.go:414] "containerMap: removing container" podUID="1b0ae648-a85e-4a86-98f1-db2e9600c67a" containerName="insights-gathering"
Oct 31 19:21:22.364301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364299    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0ae648-a85e-4a86-98f1-db2e9600c67a" containerName="insights-gathering"
Oct 31 19:21:22.364504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364371    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="54073420-a941-4d94-b5db-512984d9597b" containerName="insights-gathering"
Oct 31 19:21:22.364504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364379    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae953c47-79ec-4415-9f33-d4fbb364d6c6" containerName="registry"
Oct 31 19:21:22.364504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364385    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f1221c-9c54-4098-a301-ece68db990eb" containerName="registry"
Oct 31 19:21:22.364504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.364392    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0ae648-a85e-4a86-98f1-db2e9600c67a" containerName="insights-gathering"
Oct 31 19:21:22.367308 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.367291    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.416047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.416005    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b98296-1596-4837-8587-18e6ca94599b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.416288 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.416089    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.416288 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.416226    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.416288 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.416267    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.416423 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.416346    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.416423 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.416379    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.416423 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.416395    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.416423 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.416417    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzwm\" (UniqueName: \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-kube-api-access-hbzwm\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.416597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.416552    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b98296-1596-4837-8587-18e6ca94599b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.420177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.420145    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 19:21:22.445350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.445312    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5cqm6\""
Oct 31 19:21:22.472025 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.471988    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"]
Oct 31 19:21:22.517907 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.517802    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518083 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.517941    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518083 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.517966    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518083 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.517987    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518083 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.518010    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518323 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.518158    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzwm\" (UniqueName: \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-kube-api-access-hbzwm\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518323 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.518263    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b98296-1596-4837-8587-18e6ca94599b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518323 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.518289    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b98296-1596-4837-8587-18e6ca94599b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518480 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.518355    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518480 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.518411    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518480 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.518470    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.518482    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.518742 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.518717    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.519027 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.519005    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b98296-1596-4837-8587-18e6ca94599b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.520427 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.520406    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.520711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.520686    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b98296-1596-4837-8587-18e6ca94599b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.539933 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.539902    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:22.576194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.576153    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.576194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.576177    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzwm\" (UniqueName: \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-kube-api-access-hbzwm\") pod \"maas-default-gateway-openshift-default-58d898bd55-ks646\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.679391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.679312    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:22.681070 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.681039    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"]
Oct 31 19:21:22.684174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.684151    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.720496 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.720445    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.720732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.720535    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.720732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.720586    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.721375 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.720677    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0383d77-c63f-4507-aa64-b080e2a16a00-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.721492 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.721415    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55c5n\" (UniqueName: \"kubernetes.io/projected/b0383d77-c63f-4507-aa64-b080e2a16a00-kube-api-access-55c5n\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.721492 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.721468    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.721590 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.721509    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.721590 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.721540    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.721780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.721598    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.743779 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.743742    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"]
Oct 31 19:21:22.744385 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.744361    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-xxqxh\""
Oct 31 19:21:22.822449 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.822413    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.822621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.822455    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.822621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.822484    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.822621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.822521    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0383d77-c63f-4507-aa64-b080e2a16a00-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.822621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.822541    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55c5n\" (UniqueName: \"kubernetes.io/projected/b0383d77-c63f-4507-aa64-b080e2a16a00-kube-api-access-55c5n\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.822621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.822579    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.822621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.822612    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.822987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.822633    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.822987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.822774    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.822987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.822899    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.823142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.823038    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.823142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.823090    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.823290 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.823268    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0383d77-c63f-4507-aa64-b080e2a16a00-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.823355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.823322    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.824983 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.824961    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.825205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.825188    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.867863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.867824    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:22.880356 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.880321    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0383d77-c63f-4507-aa64-b080e2a16a00-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:22.880737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:22.880716    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55c5n\" (UniqueName: \"kubernetes.io/projected/b0383d77-c63f-4507-aa64-b080e2a16a00-kube-api-access-55c5n\") pod \"openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr\" (UID: \"b0383d77-c63f-4507-aa64-b080e2a16a00\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:23.006579 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:23.006498    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:23.090054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:23.090019    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:23.108455 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:21:23.108419    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b98296_1596_4837_8587_18e6ca94599b.slice/crio-fe8af90211ce357c51271c43b608ba6f327665841cff41237b0ca6fbfc7a1667 WatchSource:0}: Error finding container fe8af90211ce357c51271c43b608ba6f327665841cff41237b0ca6fbfc7a1667: Status 404 returned error can't find the container with id fe8af90211ce357c51271c43b608ba6f327665841cff41237b0ca6fbfc7a1667
Oct 31 19:21:23.108602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:23.108460    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"]
Oct 31 19:21:23.110492 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:23.110471    5006 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:21:23.375619 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:21:23.375575    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0383d77_c63f_4507_aa64_b080e2a16a00.slice/crio-f80a20a8c6629899479855be060b23045a21b822d53c8b929f9966d8c2af05c3 WatchSource:0}: Error finding container f80a20a8c6629899479855be060b23045a21b822d53c8b929f9966d8c2af05c3: Status 404 returned error can't find the container with id f80a20a8c6629899479855be060b23045a21b822d53c8b929f9966d8c2af05c3
Oct 31 19:21:23.378375 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:23.378346    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"]
Oct 31 19:21:24.013995 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:24.013954    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" event={"ID":"31b98296-1596-4837-8587-18e6ca94599b","Type":"ContainerStarted","Data":"fe8af90211ce357c51271c43b608ba6f327665841cff41237b0ca6fbfc7a1667"}
Oct 31 19:21:24.015104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:24.015078    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr" event={"ID":"b0383d77-c63f-4507-aa64-b080e2a16a00","Type":"ContainerStarted","Data":"f80a20a8c6629899479855be060b23045a21b822d53c8b929f9966d8c2af05c3"}
Oct 31 19:21:26.514160 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:26.514116    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:27.678014 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:27.677974    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:21:27.678339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:27.678079    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:21:27.678339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:27.678135    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:21:27.931924 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:27.931877    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:21:27.932047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:27.931954    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:21:27.932047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:27.931985    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:21:28.036645 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:28.036600    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr" event={"ID":"b0383d77-c63f-4507-aa64-b080e2a16a00","Type":"ContainerStarted","Data":"bb5ba7c171bd2a18e099ebf9efa03d377d19271420cc522ae4349367e504860f"}
Oct 31 19:21:28.038508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:28.038474    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" event={"ID":"31b98296-1596-4837-8587-18e6ca94599b","Type":"ContainerStarted","Data":"4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288"}
Oct 31 19:21:28.160006 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:28.159946    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr" podStartSLOduration=1.860782309 podStartE2EDuration="6.159929357s" totalImagesPullingTime="4.299147048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:21:28.158036509 +0000 UTC m=+306.687570562" watchObservedRunningTime="2026-10-31 19:21:28.159929357 +0000 UTC m=+306.689463409"
Oct 31 19:21:28.276024 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:28.275923    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" podStartSLOduration=1.455432072 podStartE2EDuration="6.275905773s" totalImagesPullingTime="4.820473701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:21:28.274807591 +0000 UTC m=+306.804341664" watchObservedRunningTime="2026-10-31 19:21:28.275905773 +0000 UTC m=+306.805439828"
Oct 31 19:21:28.679668 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:28.679626    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:28.684401 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:28.684366    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:29.007442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:29.007359    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:29.012207 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:29.012175    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:29.043438 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:29.043404    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:29.043636 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:29.043461    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:29.044211 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:29.044186    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr"
Oct 31 19:21:29.044413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:29.044400    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:21:35.393434 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:21:35.393354    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:21:54.576020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:54.575989    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:54.827962 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:54.827873    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:55.284913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.284875    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:55.433149 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.433114    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"]
Oct 31 19:21:55.437154 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.437127    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.492586 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.492546    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7fffj\""
Oct 31 19:21:55.510903 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.510868    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/847dc9eb-0752-43fa-8985-06e4dfc33efb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.510903 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.510907    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/847dc9eb-0752-43fa-8985-06e4dfc33efb-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.511150 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.511038    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/847dc9eb-0752-43fa-8985-06e4dfc33efb-manifests\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.511204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.511142    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6svd\" (UniqueName: \"kubernetes.io/projected/847dc9eb-0752-43fa-8985-06e4dfc33efb-kube-api-access-w6svd\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.511204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.511186    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/847dc9eb-0752-43fa-8985-06e4dfc33efb-metrics-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.511305 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.511256    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847dc9eb-0752-43fa-8985-06e4dfc33efb-tmp\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.557900 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.557810    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"]
Oct 31 19:21:55.558236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.558216    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:55.573254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.573225    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 19:21:55.573602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.573581    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 19:21:55.573985 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.573970    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 19:21:55.574278 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.574262    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 19:21:55.574342 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.574294    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 19:21:55.611670 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.611614    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6svd\" (UniqueName: \"kubernetes.io/projected/847dc9eb-0752-43fa-8985-06e4dfc33efb-kube-api-access-w6svd\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.612145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.611682    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/847dc9eb-0752-43fa-8985-06e4dfc33efb-metrics-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.612145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.611707    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847dc9eb-0752-43fa-8985-06e4dfc33efb-tmp\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.612145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.611726    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/847dc9eb-0752-43fa-8985-06e4dfc33efb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.612145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.611746    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/847dc9eb-0752-43fa-8985-06e4dfc33efb-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.612145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.611795    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/847dc9eb-0752-43fa-8985-06e4dfc33efb-manifests\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.612379 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.612272    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/847dc9eb-0752-43fa-8985-06e4dfc33efb-manifests\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.614419 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.614394    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/847dc9eb-0752-43fa-8985-06e4dfc33efb-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.614545 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.614425    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/847dc9eb-0752-43fa-8985-06e4dfc33efb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.614611 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.614572    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/847dc9eb-0752-43fa-8985-06e4dfc33efb-tmp\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.614695 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.614676    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/847dc9eb-0752-43fa-8985-06e4dfc33efb-metrics-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.676257 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.676216    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6svd\" (UniqueName: \"kubernetes.io/projected/847dc9eb-0752-43fa-8985-06e4dfc33efb-kube-api-access-w6svd\") pod \"opendatahub-operator-controller-manager-dc8db5648-zvq6r\" (UID: \"847dc9eb-0752-43fa-8985-06e4dfc33efb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.752793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.752758    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:21:55.817053 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:55.816937    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:56.072851 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:56.072769    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:56.250392 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:21:56.250354    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847dc9eb_0752_43fa_8985_06e4dfc33efb.slice/crio-c927cc75804b8b4ca0e311353e51584110095e8b2a26dc42c0c3650c6e779797 WatchSource:0}: Error finding container c927cc75804b8b4ca0e311353e51584110095e8b2a26dc42c0c3650c6e779797: Status 404 returned error can't find the container with id c927cc75804b8b4ca0e311353e51584110095e8b2a26dc42c0c3650c6e779797
Oct 31 19:21:56.259050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:56.259021    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"]
Oct 31 19:21:56.420027 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:56.419987    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:56.664579 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:56.664542    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:57.159168 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:57.159134    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r" event={"ID":"847dc9eb-0752-43fa-8985-06e4dfc33efb","Type":"ContainerStarted","Data":"c927cc75804b8b4ca0e311353e51584110095e8b2a26dc42c0c3650c6e779797"}
Oct 31 19:21:57.959172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:57.959110    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:58.272850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:58.272731    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:58.594155 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:58.594120    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:58.820286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:21:58.820182    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:00.174160 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:00.174122    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r" event={"ID":"847dc9eb-0752-43fa-8985-06e4dfc33efb","Type":"ContainerStarted","Data":"f84ab3d1a39381e641c0002a160c700e634fbfe9f85dd526b76c5fbe03ab4607"}
Oct 31 19:22:01.180067 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:01.180032    5006 generic.go:356] "Generic (PLEG): container finished" podID="847dc9eb-0752-43fa-8985-06e4dfc33efb" containerID="f84ab3d1a39381e641c0002a160c700e634fbfe9f85dd526b76c5fbe03ab4607" exitCode=0
Oct 31 19:22:01.180473 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:01.180075    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r" event={"ID":"847dc9eb-0752-43fa-8985-06e4dfc33efb","Type":"ContainerDied","Data":"f84ab3d1a39381e641c0002a160c700e634fbfe9f85dd526b76c5fbe03ab4607"}
Oct 31 19:22:02.185922 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:02.185886    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r" event={"ID":"847dc9eb-0752-43fa-8985-06e4dfc33efb","Type":"ContainerStarted","Data":"7998ba9554c83444d80e7e6154347051db13b37665e9ec024a140d056b36eb19"}
Oct 31 19:22:02.186333 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:02.186008    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:22:02.431636 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:02.431571    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r" podStartSLOduration=2.58325177 podStartE2EDuration="7.431556224s" totalImagesPullingTime="4.378548627s" totalInitContainerRuntime="469.755827ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:02.380289801 +0000 UTC m=+340.909823872" watchObservedRunningTime="2026-10-31 19:22:02.431556224 +0000 UTC m=+340.961090278"
Oct 31 19:22:04.182189 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:04.182145    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:04.408509 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:04.408474    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:05.067397 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:05.067356    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:05.315634 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:05.315598    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:05.393935 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:22:05.393894    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:22:06.931877 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:06.931839    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:08.129530 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:08.129492    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:08.968908 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:08.968868    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:09.363327 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:09.363290    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:11.377076 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:11.377043    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:12.507120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:12.507089    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:12.759752 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:12.759642    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:13.193470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:13.193430    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zvq6r"
Oct 31 19:22:13.532526 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:13.532409    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:15.074920 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:15.074879    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:15.362817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:15.362783    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:16.504368 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:16.504325    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:18.365900 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.365861    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4"]
Oct 31 19:22:18.372399 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.372374    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4"
Oct 31 19:22:18.410594 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.410556    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvql8\" (UniqueName: \"kubernetes.io/projected/3b1eee4c-644d-41e7-b954-ec9f227b53dc-kube-api-access-lvql8\") pod \"dns-operator-controller-manager-8557cbdb84-mzlv4\" (UID: \"3b1eee4c-644d-41e7-b954-ec9f227b53dc\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4"
Oct 31 19:22:18.480409 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.480372    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rv8z2\""
Oct 31 19:22:18.480409 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.480380    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 19:22:18.480409 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.480377    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 19:22:18.480409 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.480405    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 19:22:18.481882 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.481860    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4"]
Oct 31 19:22:18.511055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.511013    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvql8\" (UniqueName: \"kubernetes.io/projected/3b1eee4c-644d-41e7-b954-ec9f227b53dc-kube-api-access-lvql8\") pod \"dns-operator-controller-manager-8557cbdb84-mzlv4\" (UID: \"3b1eee4c-644d-41e7-b954-ec9f227b53dc\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4"
Oct 31 19:22:18.573260 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.573224    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvql8\" (UniqueName: \"kubernetes.io/projected/3b1eee4c-644d-41e7-b954-ec9f227b53dc-kube-api-access-lvql8\") pod \"dns-operator-controller-manager-8557cbdb84-mzlv4\" (UID: \"3b1eee4c-644d-41e7-b954-ec9f227b53dc\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4"
Oct 31 19:22:18.686301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.686208    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4"
Oct 31 19:22:18.924012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:18.923980    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:19.060840 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:22:19.060792    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1eee4c_644d_41e7_b954_ec9f227b53dc.slice/crio-b5fde9ce559bc920bcacc716280283a9afde782677a5871ae594a381c31d31e5 WatchSource:0}: Error finding container b5fde9ce559bc920bcacc716280283a9afde782677a5871ae594a381c31d31e5: Status 404 returned error can't find the container with id b5fde9ce559bc920bcacc716280283a9afde782677a5871ae594a381c31d31e5
Oct 31 19:22:19.061949 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:19.061831    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4"]
Oct 31 19:22:19.266892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:19.266858    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4" event={"ID":"3b1eee4c-644d-41e7-b954-ec9f227b53dc","Type":"ContainerStarted","Data":"b5fde9ce559bc920bcacc716280283a9afde782677a5871ae594a381c31d31e5"}
Oct 31 19:22:19.446014 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:19.445976    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:19.711561 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:19.711477    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:19.979910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:19.979826    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:20.780007 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:20.779971    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:21.020680 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:21.020630    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:21.284744 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:21.284702    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:22.196396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.196352    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:22.479491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.479384    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k26bt"]
Oct 31 19:22:22.484510 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.484467    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k26bt"
Oct 31 19:22:22.531493 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.531457    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k26bt"]
Oct 31 19:22:22.544755 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.544718    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txbc\" (UniqueName: \"kubernetes.io/projected/e3cbeb55-7b7e-462f-887f-c88b6a6f4b59-kube-api-access-2txbc\") pod \"authorino-operator-7b7bb4bc77-k26bt\" (UID: \"e3cbeb55-7b7e-462f-887f-c88b6a6f4b59\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k26bt"
Oct 31 19:22:22.580513 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.580491    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-t8j9b\""
Oct 31 19:22:22.633732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.633696    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 19:22:22.645194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.645172    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txbc\" (UniqueName: \"kubernetes.io/projected/e3cbeb55-7b7e-462f-887f-c88b6a6f4b59-kube-api-access-2txbc\") pod \"authorino-operator-7b7bb4bc77-k26bt\" (UID: \"e3cbeb55-7b7e-462f-887f-c88b6a6f4b59\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k26bt"
Oct 31 19:22:22.704106 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.704073    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txbc\" (UniqueName: \"kubernetes.io/projected/e3cbeb55-7b7e-462f-887f-c88b6a6f4b59-kube-api-access-2txbc\") pod \"authorino-operator-7b7bb4bc77-k26bt\" (UID: \"e3cbeb55-7b7e-462f-887f-c88b6a6f4b59\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k26bt"
Oct 31 19:22:22.792965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.792836    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:22.801840 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:22.801794    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k26bt"
Oct 31 19:22:23.179856 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:23.179820    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k26bt"]
Oct 31 19:22:23.208618 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:22:23.208562    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cbeb55_7b7e_462f_887f_c88b6a6f4b59.slice/crio-3d3f1fefc948544ade8c3f25eb0637c5162e58fa938c5d595a40edbe2b2b9be3 WatchSource:0}: Error finding container 3d3f1fefc948544ade8c3f25eb0637c5162e58fa938c5d595a40edbe2b2b9be3: Status 404 returned error can't find the container with id 3d3f1fefc948544ade8c3f25eb0637c5162e58fa938c5d595a40edbe2b2b9be3
Oct 31 19:22:23.286638 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:23.286602    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4" event={"ID":"3b1eee4c-644d-41e7-b954-ec9f227b53dc","Type":"ContainerStarted","Data":"fd513a7569dfe1c074407e9ecd73aea7c62b10ca35cc25e9ae0c2b0de68c4431"}
Oct 31 19:22:23.286848 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:23.286806    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4"
Oct 31 19:22:23.288079 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:23.288055    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k26bt" event={"ID":"e3cbeb55-7b7e-462f-887f-c88b6a6f4b59","Type":"ContainerStarted","Data":"3d3f1fefc948544ade8c3f25eb0637c5162e58fa938c5d595a40edbe2b2b9be3"}
Oct 31 19:22:23.407061 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:23.407001    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4" podStartSLOduration=1.8893667170000001 podStartE2EDuration="5.406983603s" totalImagesPullingTime="3.517616886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:23.405221287 +0000 UTC m=+361.934755359" watchObservedRunningTime="2026-10-31 19:22:23.406983603 +0000 UTC m=+361.936517655"
Oct 31 19:22:23.737957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:23.737915    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:24.788275 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:24.788238    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:25.642754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:25.642706    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:25.900007 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:25.899921    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:25.934396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:25.934358    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k"]
Oct 31 19:22:25.937901 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:25.937880    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k"
Oct 31 19:22:25.976826 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:25.976790    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbb64\" (UniqueName: \"kubernetes.io/projected/68655beb-12b2-4ba2-a31f-7542299aeda4-kube-api-access-fbb64\") pod \"limitador-operator-controller-manager-5778bb4449-zxk9k\" (UID: \"68655beb-12b2-4ba2-a31f-7542299aeda4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k"
Oct 31 19:22:26.045827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:26.045780    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-h65ch\""
Oct 31 19:22:26.047389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:26.047359    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k"]
Oct 31 19:22:26.078400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:26.078357    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbb64\" (UniqueName: \"kubernetes.io/projected/68655beb-12b2-4ba2-a31f-7542299aeda4-kube-api-access-fbb64\") pod \"limitador-operator-controller-manager-5778bb4449-zxk9k\" (UID: \"68655beb-12b2-4ba2-a31f-7542299aeda4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k"
Oct 31 19:22:26.136002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:26.135965    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbb64\" (UniqueName: \"kubernetes.io/projected/68655beb-12b2-4ba2-a31f-7542299aeda4-kube-api-access-fbb64\") pod \"limitador-operator-controller-manager-5778bb4449-zxk9k\" (UID: \"68655beb-12b2-4ba2-a31f-7542299aeda4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k"
Oct 31 19:22:26.250988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:26.250879    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k"
Oct 31 19:22:26.305871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:26.305845    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k26bt" event={"ID":"e3cbeb55-7b7e-462f-887f-c88b6a6f4b59","Type":"ContainerStarted","Data":"eefc397bcebafa6349a05818933c94902a309ee83af2e129005cb1914fa7e406"}
Oct 31 19:22:26.306015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:26.305995    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k26bt"
Oct 31 19:22:26.423235 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:26.423179    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k26bt" podStartSLOduration=1.8855591189999998 podStartE2EDuration="4.423163629s" totalImagesPullingTime="2.53760451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:26.422172485 +0000 UTC m=+364.951706558" watchObservedRunningTime="2026-10-31 19:22:26.423163629 +0000 UTC m=+364.952697682"
Oct 31 19:22:26.618073 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:22:26.618043    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68655beb_12b2_4ba2_a31f_7542299aeda4.slice/crio-8ae1c7fad354afeb1b1b033cd9db65feb874da1ef2a6e74dfbc1a63f41a170ea WatchSource:0}: Error finding container 8ae1c7fad354afeb1b1b033cd9db65feb874da1ef2a6e74dfbc1a63f41a170ea: Status 404 returned error can't find the container with id 8ae1c7fad354afeb1b1b033cd9db65feb874da1ef2a6e74dfbc1a63f41a170ea
Oct 31 19:22:26.619538 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:26.619506    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k"]
Oct 31 19:22:27.311989 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:27.311939    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k" event={"ID":"68655beb-12b2-4ba2-a31f-7542299aeda4","Type":"ContainerStarted","Data":"8ae1c7fad354afeb1b1b033cd9db65feb874da1ef2a6e74dfbc1a63f41a170ea"}
Oct 31 19:22:27.514472 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:27.514432    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:27.742634 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:27.742596    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:28.039789 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:28.039635    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:28.279082 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:28.279040    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:28.687867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:28.687832    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:29.775777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:29.775734    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:30.021570 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.021537    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:30.253113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.253077    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:30.541686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.541545    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"]
Oct 31 19:22:30.553837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.553801    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:30.591361 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.591322    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.605694 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.605664    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"]
Oct 31 19:22:30.620366 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.620331    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.620366 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.620376    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.620572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.620405    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.620572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.620523    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmmr\" (UniqueName: \"kubernetes.io/projected/2685eee1-8b37-4d0b-8003-8e85186cc299-kube-api-access-rkmmr\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.620663 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.620572    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-token\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.620705 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.620665    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.620738 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.620704    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685eee1-8b37-4d0b-8003-8e85186cc299-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.620738 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.620724    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.620803 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.620746    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-data\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.654101 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.654064    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-cgbjp\""
Oct 31 19:22:30.722228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722187    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmmr\" (UniqueName: \"kubernetes.io/projected/2685eee1-8b37-4d0b-8003-8e85186cc299-kube-api-access-rkmmr\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722246    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-token\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722279    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722306    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685eee1-8b37-4d0b-8003-8e85186cc299-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722325    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722353    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-data\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722513    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722584    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722627    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722876 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722821    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-data\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722892    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.722931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.722903    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.723102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.723000    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.723279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.723255    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2685eee1-8b37-4d0b-8003-8e85186cc299-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.724713 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.724693    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.725066 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.725050    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.783141 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.783103    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmmr\" (UniqueName: \"kubernetes.io/projected/2685eee1-8b37-4d0b-8003-8e85186cc299-kube-api-access-rkmmr\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.786670 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.786622    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2685eee1-8b37-4d0b-8003-8e85186cc299-istio-token\") pod \"data-science-gateway-data-science-gateway-class-69f549cf5c44lqr\" (UID: \"2685eee1-8b37-4d0b-8003-8e85186cc299\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.813643 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.813553    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:30.864725 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.864682    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-qc5jm"]
Oct 31 19:22:30.888880 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.888843    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-qc5jm"
Oct 31 19:22:30.924082 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.924052    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2262f176-01ff-4ea3-8566-edea21934ee6-bound-sa-token\") pod \"cert-manager-597b95cb84-qc5jm\" (UID: \"2262f176-01ff-4ea3-8566-edea21934ee6\") " pod="cert-manager/cert-manager-597b95cb84-qc5jm"
Oct 31 19:22:30.924276 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.924161    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9kk\" (UniqueName: \"kubernetes.io/projected/2262f176-01ff-4ea3-8566-edea21934ee6-kube-api-access-wk9kk\") pod \"cert-manager-597b95cb84-qc5jm\" (UID: \"2262f176-01ff-4ea3-8566-edea21934ee6\") " pod="cert-manager/cert-manager-597b95cb84-qc5jm"
Oct 31 19:22:30.925155 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.925128    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-qc5jm"]
Oct 31 19:22:30.933667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.933617    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:30.942187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.942152    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 19:22:30.942365 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.942278    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ln2kk\""
Oct 31 19:22:30.942365 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:30.942280    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 19:22:31.025208 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.025170    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9kk\" (UniqueName: \"kubernetes.io/projected/2262f176-01ff-4ea3-8566-edea21934ee6-kube-api-access-wk9kk\") pod \"cert-manager-597b95cb84-qc5jm\" (UID: \"2262f176-01ff-4ea3-8566-edea21934ee6\") " pod="cert-manager/cert-manager-597b95cb84-qc5jm"
Oct 31 19:22:31.025381 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.025255    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2262f176-01ff-4ea3-8566-edea21934ee6-bound-sa-token\") pod \"cert-manager-597b95cb84-qc5jm\" (UID: \"2262f176-01ff-4ea3-8566-edea21934ee6\") " pod="cert-manager/cert-manager-597b95cb84-qc5jm"
Oct 31 19:22:31.095475 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.095430    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2262f176-01ff-4ea3-8566-edea21934ee6-bound-sa-token\") pod \"cert-manager-597b95cb84-qc5jm\" (UID: \"2262f176-01ff-4ea3-8566-edea21934ee6\") " pod="cert-manager/cert-manager-597b95cb84-qc5jm"
Oct 31 19:22:31.095475 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.095471    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9kk\" (UniqueName: \"kubernetes.io/projected/2262f176-01ff-4ea3-8566-edea21934ee6-kube-api-access-wk9kk\") pod \"cert-manager-597b95cb84-qc5jm\" (UID: \"2262f176-01ff-4ea3-8566-edea21934ee6\") " pod="cert-manager/cert-manager-597b95cb84-qc5jm"
Oct 31 19:22:31.180515 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.180477    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:31.230276 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.230240    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-qc5jm"
Oct 31 19:22:31.334159 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.334123    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k" event={"ID":"68655beb-12b2-4ba2-a31f-7542299aeda4","Type":"ContainerStarted","Data":"a3045f01c0ab67d7978d27f75fc79d5cf1471b7e58cfc062ca1ed95b671dae7e"}
Oct 31 19:22:31.334336 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.334222    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k"
Oct 31 19:22:31.466797 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.466696    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k" podStartSLOduration=1.946761427 podStartE2EDuration="6.466684092s" totalImagesPullingTime="4.519922665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:31.46621934 +0000 UTC m=+369.995753393" watchObservedRunningTime="2026-10-31 19:22:31.466684092 +0000 UTC m=+369.996218139"
Oct 31 19:22:31.601450 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:22:31.601421    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2685eee1_8b37_4d0b_8003_8e85186cc299.slice/crio-4c58fa288cc0ad85ae2d6beba2fd816b0ce926dfc1db2f29b1b65e4412aab9f3 WatchSource:0}: Error finding container 4c58fa288cc0ad85ae2d6beba2fd816b0ce926dfc1db2f29b1b65e4412aab9f3: Status 404 returned error can't find the container with id 4c58fa288cc0ad85ae2d6beba2fd816b0ce926dfc1db2f29b1b65e4412aab9f3
Oct 31 19:22:31.603625 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.603604    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"]
Oct 31 19:22:31.603723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.603643    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:22:31.603763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.603730    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:22:31.603763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.603757    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:22:31.616047 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:22:31.615983    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2262f176_01ff_4ea3_8566_edea21934ee6.slice/crio-d689e18c028834db2c05c81e8cff3ca30eed739c9690128a407da65d9df93978 WatchSource:0}: Error finding container d689e18c028834db2c05c81e8cff3ca30eed739c9690128a407da65d9df93978: Status 404 returned error can't find the container with id d689e18c028834db2c05c81e8cff3ca30eed739c9690128a407da65d9df93978
Oct 31 19:22:31.616876 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:31.616856    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-qc5jm"]
Oct 31 19:22:32.341334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:32.341300    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr" event={"ID":"2685eee1-8b37-4d0b-8003-8e85186cc299","Type":"ContainerStarted","Data":"2b6b4acbb94bae19dc4f657827316b1da19f9e99f5f728f8ea682b0821a39fe3"}
Oct 31 19:22:32.341833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:32.341392    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr" event={"ID":"2685eee1-8b37-4d0b-8003-8e85186cc299","Type":"ContainerStarted","Data":"4c58fa288cc0ad85ae2d6beba2fd816b0ce926dfc1db2f29b1b65e4412aab9f3"}
Oct 31 19:22:32.342704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:32.342674    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-qc5jm" event={"ID":"2262f176-01ff-4ea3-8566-edea21934ee6","Type":"ContainerStarted","Data":"d689e18c028834db2c05c81e8cff3ca30eed739c9690128a407da65d9df93978"}
Oct 31 19:22:32.934862 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:32.934813    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:32.939988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:32.939957    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:33.060385 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:33.060316    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr" podStartSLOduration=3.060299304 podStartE2EDuration="3.060299304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:32.470079034 +0000 UTC m=+370.999613105" watchObservedRunningTime="2026-10-31 19:22:33.060299304 +0000 UTC m=+371.589833436"
Oct 31 19:22:33.349477 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:33.349408    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:33.350644 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:33.350615    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c44lqr"
Oct 31 19:22:34.295467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:34.295433    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mzlv4"
Oct 31 19:22:35.398752 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:22:35.398710    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:22:36.181587 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:36.181498    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:36.373086 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:36.373044    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-qc5jm" event={"ID":"2262f176-01ff-4ea3-8566-edea21934ee6","Type":"ContainerStarted","Data":"93969688b095dd1e19632d64b4d5e211ce2cade42ce7edc664cee29f2e20ee04"}
Oct 31 19:22:36.455681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:36.455591    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:36.494029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:36.493972    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-qc5jm" podStartSLOduration=2.266424721 podStartE2EDuration="6.49395962s" totalImagesPullingTime="4.227534899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:36.493627453 +0000 UTC m=+375.023161502" watchObservedRunningTime="2026-10-31 19:22:36.49395962 +0000 UTC m=+375.023493672"
Oct 31 19:22:37.314695 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:37.314641    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k26bt"
Oct 31 19:22:37.478999 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:37.478968    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:42.345615 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:42.345571    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zxk9k"
Oct 31 19:22:42.768804 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:42.768719    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:43.088166 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:43.088134    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:45.736180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:45.736142    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:45.954878 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:45.954848    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:50.743398 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:50.743362    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:50.974955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:50.974913    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:51.189211 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:51.189176    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:51.525399 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:51.525317    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:51.755850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:51.755819    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:54.902263 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:54.902228    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"]
Oct 31 19:22:54.907961 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:54.907936    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:54.961064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:54.961027    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 19:22:54.961064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:54.961027    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 19:22:55.060441 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.060402    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d53b3837-5749-40ea-956b-344928900061-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.060643 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.060468    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lljqs\" (UniqueName: \"kubernetes.io/projected/d53b3837-5749-40ea-956b-344928900061-kube-api-access-lljqs\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.060643 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.060544    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d53b3837-5749-40ea-956b-344928900061-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.060643 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.060567    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d53b3837-5749-40ea-956b-344928900061-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.112936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.112865    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-lx64s\""
Oct 31 19:22:55.113160 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.113034    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 31 19:22:55.114898 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.114870    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"]
Oct 31 19:22:55.161727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.161612    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lljqs\" (UniqueName: \"kubernetes.io/projected/d53b3837-5749-40ea-956b-344928900061-kube-api-access-lljqs\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.161727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.161695    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d53b3837-5749-40ea-956b-344928900061-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.161727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.161717    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d53b3837-5749-40ea-956b-344928900061-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.161951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.161775    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d53b3837-5749-40ea-956b-344928900061-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.162118 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.162099    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d53b3837-5749-40ea-956b-344928900061-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.162156 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.162144    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d53b3837-5749-40ea-956b-344928900061-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.162792 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.162774    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d53b3837-5749-40ea-956b-344928900061-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.219627 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.219588    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lljqs\" (UniqueName: \"kubernetes.io/projected/d53b3837-5749-40ea-956b-344928900061-kube-api-access-lljqs\") pod \"kserve-module-controller-manager-7566cf5fc8-pfnsg\" (UID: \"d53b3837-5749-40ea-956b-344928900061\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.221255 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.221234    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:55.637222 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:22:55.637185    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53b3837_5749_40ea_956b_344928900061.slice/crio-2c31d10bd134ffbbe25a2626abac2f52d9c04967e971278883787f45a1628aa9 WatchSource:0}: Error finding container 2c31d10bd134ffbbe25a2626abac2f52d9c04967e971278883787f45a1628aa9: Status 404 returned error can't find the container with id 2c31d10bd134ffbbe25a2626abac2f52d9c04967e971278883787f45a1628aa9
Oct 31 19:22:55.639147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:55.639119    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"]
Oct 31 19:22:56.470663 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:56.470625    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg" event={"ID":"d53b3837-5749-40ea-956b-344928900061","Type":"ContainerStarted","Data":"2c31d10bd134ffbbe25a2626abac2f52d9c04967e971278883787f45a1628aa9"}
Oct 31 19:22:56.946047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:56.946011    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:57.229720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:57.229596    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:57.583341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:57.583256    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:57.791833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:57.791735    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:59.487959 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:59.487924    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg" event={"ID":"d53b3837-5749-40ea-956b-344928900061","Type":"ContainerStarted","Data":"528cef550819efe56d5641549296008b636a2d030380462155e0cc9cd52a3f46"}
Oct 31 19:22:59.488356 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:59.488051    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:22:59.656771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:59.656703    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg" podStartSLOduration=2.701550617 podStartE2EDuration="5.656681417s" totalImagesPullingTime="2.9551308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:59.65434086 +0000 UTC m=+398.183874912" watchObservedRunningTime="2026-10-31 19:22:59.656681417 +0000 UTC m=+398.186215463"
Oct 31 19:22:59.886381 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:22:59.886349    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:03.431284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:23:03.431184    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:03.650960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:23:03.650915    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:04.157071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:23:04.157039    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:04.375075 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:23:04.375029    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:05.393850 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:23:05.393812    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:23:07.144761 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:23:07.144722    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:07.366438 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:23:07.366396    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:13.848383 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:23:13.848344    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:20.493967 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:23:20.493934    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pfnsg"
Oct 31 19:23:35.392116 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:23:35.392074    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:23:49.975382 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:23:49.975339    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:01.135190 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:01.135155    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:01.353236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:01.353203    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:01.597164 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:01.597117    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:01.816235 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:01.816183    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:02.040776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.040634    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:02.051013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.050980    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"]
Oct 31 19:24:02.058306 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.058279    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:24:02.109234 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.109196    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"]
Oct 31 19:24:02.147272 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.147231    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7c16946-8cf8-4974-912d-43a90d6371e0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-g85kp\" (UID: \"a7c16946-8cf8-4974-912d-43a90d6371e0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:24:02.147701 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.147279    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc562\" (UniqueName: \"kubernetes.io/projected/a7c16946-8cf8-4974-912d-43a90d6371e0-kube-api-access-gc562\") pod \"llmisvc-controller-manager-869d5dcc5d-g85kp\" (UID: \"a7c16946-8cf8-4974-912d-43a90d6371e0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:24:02.157055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.157017    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-k9fw7\""
Oct 31 19:24:02.161092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.161064    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 31 19:24:02.248117 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.248079    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7c16946-8cf8-4974-912d-43a90d6371e0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-g85kp\" (UID: \"a7c16946-8cf8-4974-912d-43a90d6371e0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:24:02.248117 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.248119    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc562\" (UniqueName: \"kubernetes.io/projected/a7c16946-8cf8-4974-912d-43a90d6371e0-kube-api-access-gc562\") pod \"llmisvc-controller-manager-869d5dcc5d-g85kp\" (UID: \"a7c16946-8cf8-4974-912d-43a90d6371e0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:24:02.250732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.250703    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7c16946-8cf8-4974-912d-43a90d6371e0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-g85kp\" (UID: \"a7c16946-8cf8-4974-912d-43a90d6371e0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:24:02.265597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.265559    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:02.285350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.285309    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-5bdnl"]
Oct 31 19:24:02.290432 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.290397    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:02.305102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.305071    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc562\" (UniqueName: \"kubernetes.io/projected/a7c16946-8cf8-4974-912d-43a90d6371e0-kube-api-access-gc562\") pod \"llmisvc-controller-manager-869d5dcc5d-g85kp\" (UID: \"a7c16946-8cf8-4974-912d-43a90d6371e0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:24:02.341398 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.341360    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5bdnl"]
Oct 31 19:24:02.343696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.343667    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wb7zz\""
Oct 31 19:24:02.343877 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.343680    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 31 19:24:02.343877 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.343735    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 31 19:24:02.371185 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.371143    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:24:02.450282 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.450242    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ed068b-24d8-49f9-a605-66d9499b2e4f-metrics-certs\") pod \"odh-model-controller-769875db56-5bdnl\" (UID: \"51ed068b-24d8-49f9-a605-66d9499b2e4f\") " pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:02.450443 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.450378    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51ed068b-24d8-49f9-a605-66d9499b2e4f-cert\") pod \"odh-model-controller-769875db56-5bdnl\" (UID: \"51ed068b-24d8-49f9-a605-66d9499b2e4f\") " pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:02.450486 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.450440    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24ntq\" (UniqueName: \"kubernetes.io/projected/51ed068b-24d8-49f9-a605-66d9499b2e4f-kube-api-access-24ntq\") pod \"odh-model-controller-769875db56-5bdnl\" (UID: \"51ed068b-24d8-49f9-a605-66d9499b2e4f\") " pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:02.551097 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.551009    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24ntq\" (UniqueName: \"kubernetes.io/projected/51ed068b-24d8-49f9-a605-66d9499b2e4f-kube-api-access-24ntq\") pod \"odh-model-controller-769875db56-5bdnl\" (UID: \"51ed068b-24d8-49f9-a605-66d9499b2e4f\") " pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:02.551097 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.551049    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ed068b-24d8-49f9-a605-66d9499b2e4f-metrics-certs\") pod \"odh-model-controller-769875db56-5bdnl\" (UID: \"51ed068b-24d8-49f9-a605-66d9499b2e4f\") " pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:02.551334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.551110    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51ed068b-24d8-49f9-a605-66d9499b2e4f-cert\") pod \"odh-model-controller-769875db56-5bdnl\" (UID: \"51ed068b-24d8-49f9-a605-66d9499b2e4f\") " pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:02.553694 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.553665    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51ed068b-24d8-49f9-a605-66d9499b2e4f-cert\") pod \"odh-model-controller-769875db56-5bdnl\" (UID: \"51ed068b-24d8-49f9-a605-66d9499b2e4f\") " pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:02.553827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.553803    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ed068b-24d8-49f9-a605-66d9499b2e4f-metrics-certs\") pod \"odh-model-controller-769875db56-5bdnl\" (UID: \"51ed068b-24d8-49f9-a605-66d9499b2e4f\") " pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:02.608566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.608532    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24ntq\" (UniqueName: \"kubernetes.io/projected/51ed068b-24d8-49f9-a605-66d9499b2e4f-kube-api-access-24ntq\") pod \"odh-model-controller-769875db56-5bdnl\" (UID: \"51ed068b-24d8-49f9-a605-66d9499b2e4f\") " pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:02.734422 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:24:02.734389    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda7c16946_8cf8_4974_912d_43a90d6371e0.slice/crio-9d7ce86af6e44ff37c264928179896690493e5fce5bfc5adbdef704253717ab1 WatchSource:0}: Error finding container 9d7ce86af6e44ff37c264928179896690493e5fce5bfc5adbdef704253717ab1: Status 404 returned error can't find the container with id 9d7ce86af6e44ff37c264928179896690493e5fce5bfc5adbdef704253717ab1
Oct 31 19:24:02.737134 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.737111    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"]
Oct 31 19:24:02.788959 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.788923    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp" event={"ID":"a7c16946-8cf8-4974-912d-43a90d6371e0","Type":"ContainerStarted","Data":"9d7ce86af6e44ff37c264928179896690493e5fce5bfc5adbdef704253717ab1"}
Oct 31 19:24:02.905574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:02.905532    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:03.273124 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:24:03.273087    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ed068b_24d8_49f9_a605_66d9499b2e4f.slice/crio-7c4e876d04db377648ef06e67d373bb73ae1fa8af82130aaf926814beef7dca8 WatchSource:0}: Error finding container 7c4e876d04db377648ef06e67d373bb73ae1fa8af82130aaf926814beef7dca8: Status 404 returned error can't find the container with id 7c4e876d04db377648ef06e67d373bb73ae1fa8af82130aaf926814beef7dca8
Oct 31 19:24:03.275235 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:03.275209    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5bdnl"]
Oct 31 19:24:03.378205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:03.378158    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:03.795180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:03.795142    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5bdnl" event={"ID":"51ed068b-24d8-49f9-a605-66d9499b2e4f","Type":"ContainerStarted","Data":"7c4e876d04db377648ef06e67d373bb73ae1fa8af82130aaf926814beef7dca8"}
Oct 31 19:24:05.399567 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:24:05.399471    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:24:05.810407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:05.810310    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp" event={"ID":"a7c16946-8cf8-4974-912d-43a90d6371e0","Type":"ContainerStarted","Data":"a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794"}
Oct 31 19:24:05.810574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:05.810408    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:24:05.927539 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:05.927477    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp" podStartSLOduration=1.133536725 podStartE2EDuration="3.927459317s" totalImagesPullingTime="2.793922592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:05.925185792 +0000 UTC m=+464.454719844" watchObservedRunningTime="2026-10-31 19:24:05.927459317 +0000 UTC m=+464.456993370"
Oct 31 19:24:06.816845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:06.816804    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5bdnl" event={"ID":"51ed068b-24d8-49f9-a605-66d9499b2e4f","Type":"ContainerStarted","Data":"fc8b8216de337e0093b68a0498d49ffc96c6925115e31287bb931ad96aed65b1"}
Oct 31 19:24:06.933482 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:06.933424    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-5bdnl" podStartSLOduration=1.636891559 podStartE2EDuration="4.933410126s" totalImagesPullingTime="3.296518567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:06.931897996 +0000 UTC m=+465.461432049" watchObservedRunningTime="2026-10-31 19:24:06.933410126 +0000 UTC m=+465.462944179"
Oct 31 19:24:07.821213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:07.821182    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:08.520961 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:08.520925    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:08.743981 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:08.743944    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:08.960690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:08.960641    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:09.237043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:09.236953    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:09.476897 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:09.476863    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:09.695977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:09.695938    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:09.917855 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:09.917822    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:11.906420 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:11.906377    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:14.181811 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:14.181724    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:18.829521 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:18.829482    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-5bdnl"
Oct 31 19:24:35.392435 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:24:35.392349    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:24:36.819085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:24:36.819055    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:25:05.393324 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:25:05.393287    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:25:14.515912 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:25:14.515873    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:35.392496 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:25:35.392458    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:26:05.393397 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:26:05.393288    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:26:12.710469 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:26:12.710432    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:14.871983 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:26:14.871943    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:21.636064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:26:21.636032    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:26:21.636532 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:26:21.636076    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:26:35.393307 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:26:35.393265    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:26:48.177052 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:26:48.177016    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:48.394035 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:26:48.394000    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:05.392347 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:27:05.392308    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:27:15.179868 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:27:15.179833    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:29.788450 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:27:29.788420    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:35.393768 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:27:35.393680    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:27:37.602931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:27:37.602894    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:51.926153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:27:51.926115    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:05.393379 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:28:05.393335    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:28:15.520386 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:28:15.520326    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:35.392559 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:28:35.392518    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:29:05.393786 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:29:05.393701    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:29:15.848891 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:29:15.848854    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:35.393015 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:29:35.392971    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:30:05.392947 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:30:05.392910    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:30:14.805233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:30:14.805199    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:15.020967 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:30:15.020933    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:16.186078 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:30:16.186043    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:35.393015 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:30:35.392938    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:31:05.392549 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:31:05.392503    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:31:16.446168 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:31:16.446133    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:21.664552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:31:21.664512    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:31:21.665479 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:31:21.665463    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:31:35.393841 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:31:35.393799    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:32:05.392595 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:32:05.392505    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:32:16.952741 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:32:16.952697    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:35.392803 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:32:35.392767    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:33:05.393590 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:33:05.393535    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:33:10.787996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:10.787955    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:11.106830 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:11.106793    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:11.312916 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:11.312880    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:12.231390 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.231353    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xwgjb"]
Oct 31 19:33:12.234987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.234969    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-xwgjb"
Oct 31 19:33:12.288266 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.288233    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xwgjb"]
Oct 31 19:33:12.336153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.336110    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ldbjt\""
Oct 31 19:33:12.377761 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.377724    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pw2t\" (UniqueName: \"kubernetes.io/projected/c0ff4430-a995-4820-a6a5-44c980d0288c-kube-api-access-8pw2t\") pod \"authorino-57f7f8659f-xwgjb\" (UID: \"c0ff4430-a995-4820-a6a5-44c980d0288c\") " pod="kuadrant-system/authorino-57f7f8659f-xwgjb"
Oct 31 19:33:12.468952 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.468913    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:12.479142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.479115    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pw2t\" (UniqueName: \"kubernetes.io/projected/c0ff4430-a995-4820-a6a5-44c980d0288c-kube-api-access-8pw2t\") pod \"authorino-57f7f8659f-xwgjb\" (UID: \"c0ff4430-a995-4820-a6a5-44c980d0288c\") " pod="kuadrant-system/authorino-57f7f8659f-xwgjb"
Oct 31 19:33:12.536363 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.536283    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pw2t\" (UniqueName: \"kubernetes.io/projected/c0ff4430-a995-4820-a6a5-44c980d0288c-kube-api-access-8pw2t\") pod \"authorino-57f7f8659f-xwgjb\" (UID: \"c0ff4430-a995-4820-a6a5-44c980d0288c\") " pod="kuadrant-system/authorino-57f7f8659f-xwgjb"
Oct 31 19:33:12.546006 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.545983    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-xwgjb"
Oct 31 19:33:12.681216 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.681182    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:12.911718 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:33:12.911675    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0ff4430_a995_4820_a6a5_44c980d0288c.slice/crio-126b1f7734d75090249f081c035b99f281f3dac19cae48b77af4e4fe74828f7f WatchSource:0}: Error finding container 126b1f7734d75090249f081c035b99f281f3dac19cae48b77af4e4fe74828f7f: Status 404 returned error can't find the container with id 126b1f7734d75090249f081c035b99f281f3dac19cae48b77af4e4fe74828f7f
Oct 31 19:33:12.913332 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.913304    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xwgjb"]
Oct 31 19:33:12.913441 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:12.913382    5006 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:33:13.325898 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:13.325814    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-xwgjb" event={"ID":"c0ff4430-a995-4820-a6a5-44c980d0288c","Type":"ContainerStarted","Data":"126b1f7734d75090249f081c035b99f281f3dac19cae48b77af4e4fe74828f7f"}
Oct 31 19:33:14.904249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:14.904212    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:33:14.908017 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:14.907990    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"
Oct 31 19:33:14.962028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:14.961986    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:33:15.003142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.003110    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck7f9\" (UniqueName: \"kubernetes.io/projected/aefb9b87-e1f9-47f4-ad88-17f86708bacf-kube-api-access-ck7f9\") pod \"limitador-limitador-7c9b87d574-6qj5r\" (UID: \"aefb9b87-e1f9-47f4-ad88-17f86708bacf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"
Oct 31 19:33:15.003321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.003176    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aefb9b87-e1f9-47f4-ad88-17f86708bacf-config-file\") pod \"limitador-limitador-7c9b87d574-6qj5r\" (UID: \"aefb9b87-e1f9-47f4-ad88-17f86708bacf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"
Oct 31 19:33:15.010343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.010274    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8rcsr\""
Oct 31 19:33:15.010343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.010299    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 19:33:15.104167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.104128    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck7f9\" (UniqueName: \"kubernetes.io/projected/aefb9b87-e1f9-47f4-ad88-17f86708bacf-kube-api-access-ck7f9\") pod \"limitador-limitador-7c9b87d574-6qj5r\" (UID: \"aefb9b87-e1f9-47f4-ad88-17f86708bacf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"
Oct 31 19:33:15.104351 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.104266    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aefb9b87-e1f9-47f4-ad88-17f86708bacf-config-file\") pod \"limitador-limitador-7c9b87d574-6qj5r\" (UID: \"aefb9b87-e1f9-47f4-ad88-17f86708bacf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"
Oct 31 19:33:15.105042 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.105021    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aefb9b87-e1f9-47f4-ad88-17f86708bacf-config-file\") pod \"limitador-limitador-7c9b87d574-6qj5r\" (UID: \"aefb9b87-e1f9-47f4-ad88-17f86708bacf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"
Oct 31 19:33:15.163145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.163074    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck7f9\" (UniqueName: \"kubernetes.io/projected/aefb9b87-e1f9-47f4-ad88-17f86708bacf-kube-api-access-ck7f9\") pod \"limitador-limitador-7c9b87d574-6qj5r\" (UID: \"aefb9b87-e1f9-47f4-ad88-17f86708bacf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"
Oct 31 19:33:15.221644 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.221593    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:33:15.223386 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.223352    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"
Oct 31 19:33:15.602208 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:33:15.602124    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefb9b87_e1f9_47f4_ad88_17f86708bacf.slice/crio-647d6e5c8ce025c48bee993d4685af5f957b89a08f8ee2dea31fc0f7b022974e WatchSource:0}: Error finding container 647d6e5c8ce025c48bee993d4685af5f957b89a08f8ee2dea31fc0f7b022974e: Status 404 returned error can't find the container with id 647d6e5c8ce025c48bee993d4685af5f957b89a08f8ee2dea31fc0f7b022974e
Oct 31 19:33:15.604244 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:15.604159    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:33:16.348534 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:16.348489    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-xwgjb" event={"ID":"c0ff4430-a995-4820-a6a5-44c980d0288c","Type":"ContainerStarted","Data":"20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9"}
Oct 31 19:33:16.356216 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:16.356171    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r" event={"ID":"aefb9b87-e1f9-47f4-ad88-17f86708bacf","Type":"ContainerStarted","Data":"647d6e5c8ce025c48bee993d4685af5f957b89a08f8ee2dea31fc0f7b022974e"}
Oct 31 19:33:16.461176 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:16.461114    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-xwgjb" podStartSLOduration=1.411717798 podStartE2EDuration="4.461096229s" totalImagesPullingTime="3.049378431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:16.458950442 +0000 UTC m=+1014.988484492" watchObservedRunningTime="2026-10-31 19:33:16.461096229 +0000 UTC m=+1014.990630281"
Oct 31 19:33:17.187949 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:17.187913    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:18.367170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:18.367131    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r" event={"ID":"aefb9b87-e1f9-47f4-ad88-17f86708bacf","Type":"ContainerStarted","Data":"c5965a45bb101ce87ea08edef351ea8d91edc251a9d844181129d9ad032c22f2"}
Oct 31 19:33:18.367581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:18.367198    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"
Oct 31 19:33:18.481578 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:18.481507    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r" podStartSLOduration=2.502869902 podStartE2EDuration="4.481486891s" totalImagesPullingTime="1.978616989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:18.480912822 +0000 UTC m=+1017.010446894" watchObservedRunningTime="2026-10-31 19:33:18.481486891 +0000 UTC m=+1017.011020945"
Oct 31 19:33:18.795987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:18.795899    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xwgjb"]
Oct 31 19:33:18.796138 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:18.796073    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-xwgjb" podUID="c0ff4430-a995-4820-a6a5-44c980d0288c" containerName="authorino" containerID="cri-o://20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9" gracePeriod=30
Oct 31 19:33:19.060319 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.060259    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-xwgjb"
Oct 31 19:33:19.140697 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.140634    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ff4430-a995-4820-a6a5-44c980d0288c-kube-api-access-8pw2t\" (UniqueName: \"kubernetes.io/projected/c0ff4430-a995-4820-a6a5-44c980d0288c-kube-api-access-8pw2t\") pod \"c0ff4430-a995-4820-a6a5-44c980d0288c\" (UID: \"c0ff4430-a995-4820-a6a5-44c980d0288c\") "
Oct 31 19:33:19.142910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.142883    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ff4430-a995-4820-a6a5-44c980d0288c-kube-api-access-8pw2t" pod "c0ff4430-a995-4820-a6a5-44c980d0288c" (UID: "c0ff4430-a995-4820-a6a5-44c980d0288c"). InnerVolumeSpecName "kube-api-access-8pw2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:33:19.242352 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.242315    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pw2t\" (UniqueName: \"kubernetes.io/projected/c0ff4430-a995-4820-a6a5-44c980d0288c-kube-api-access-8pw2t\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:19.373514 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.373474    5006 generic.go:356] "Generic (PLEG): container finished" podID="c0ff4430-a995-4820-a6a5-44c980d0288c" containerID="20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9" exitCode=0
Oct 31 19:33:19.373514 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.373517    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-xwgjb"
Oct 31 19:33:19.374029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.373570    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-xwgjb" event={"ID":"c0ff4430-a995-4820-a6a5-44c980d0288c","Type":"ContainerDied","Data":"20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9"}
Oct 31 19:33:19.374029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.373605    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-xwgjb" event={"ID":"c0ff4430-a995-4820-a6a5-44c980d0288c","Type":"ContainerDied","Data":"126b1f7734d75090249f081c035b99f281f3dac19cae48b77af4e4fe74828f7f"}
Oct 31 19:33:19.374029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.373626    5006 scope.go:122] "RemoveContainer" containerID="20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9"
Oct 31 19:33:19.382542 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.382524    5006 scope.go:122] "RemoveContainer" containerID="20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9"
Oct 31 19:33:19.382970 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:33:19.382950    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9\": container with ID starting with 20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9 not found: ID does not exist" containerID="20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9"
Oct 31 19:33:19.383030 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.382981    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9"} err="failed to get container status \"20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9\": rpc error: code = NotFound desc = could not find container \"20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9\": container with ID starting with 20e61407732a059f7d25b3a2a478e9c081cb073d337bd10fa798eb07493ac9c9 not found: ID does not exist"
Oct 31 19:33:19.537856 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.537813    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xwgjb"]
Oct 31 19:33:19.542725 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.542692    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xwgjb"]
Oct 31 19:33:19.597269 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:19.597235    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ff4430-a995-4820-a6a5-44c980d0288c" path="/var/lib/kubelet/pods/c0ff4430-a995-4820-a6a5-44c980d0288c/volumes"
Oct 31 19:33:29.375132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:29.375096    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"
Oct 31 19:33:32.951551 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:32.951467    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:33.166623 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:33.166583    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:33.723382 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:33.723331    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:35.393152 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:33:35.393116    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:33:54.785248 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:54.785207    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:55.007519 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:55.007466    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:55.245845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:55.245807    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:55.467590 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:55.467552    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:59.564046 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:59.564011    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:59.782966 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:33:59.782934    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:02.903548 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:02.903509    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:03.128492 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:03.128456    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:05.392317 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:34:05.392279    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:34:17.545503 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:17.545471    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:29.579476 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:29.579426    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:29.815709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:29.815678    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:31.781731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:31.781695    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:32.009246 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:32.009194    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:32.222437 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:32.222399    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:35.393139 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:34:35.393100    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:34:38.135228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:38.135140    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:34:42.641671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:42.641609    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:42.847133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:42.847085    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hrl6h"]
Oct 31 19:34:42.847526 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:42.847513    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c0ff4430-a995-4820-a6a5-44c980d0288c" containerName="authorino"
Oct 31 19:34:42.847570 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:42.847528    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ff4430-a995-4820-a6a5-44c980d0288c" containerName="authorino"
Oct 31 19:34:42.847629 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:42.847620    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ff4430-a995-4820-a6a5-44c980d0288c" containerName="authorino"
Oct 31 19:34:42.850872 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:42.850854    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hrl6h"
Oct 31 19:34:42.888808 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:42.888760    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755cq\" (UniqueName: \"kubernetes.io/projected/0d9e3e95-963d-42f5-adca-8862c7cbe16b-kube-api-access-755cq\") pod \"authorino-6f6944bcdd-hrl6h\" (UID: \"0d9e3e95-963d-42f5-adca-8862c7cbe16b\") " pod="kuadrant-system/authorino-6f6944bcdd-hrl6h"
Oct 31 19:34:42.926189 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:42.926106    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hrl6h"]
Oct 31 19:34:42.973747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:42.973702    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ldbjt\""
Oct 31 19:34:42.990157 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:42.990122    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755cq\" (UniqueName: \"kubernetes.io/projected/0d9e3e95-963d-42f5-adca-8862c7cbe16b-kube-api-access-755cq\") pod \"authorino-6f6944bcdd-hrl6h\" (UID: \"0d9e3e95-963d-42f5-adca-8862c7cbe16b\") " pod="kuadrant-system/authorino-6f6944bcdd-hrl6h"
Oct 31 19:34:43.050097 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.050063    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755cq\" (UniqueName: \"kubernetes.io/projected/0d9e3e95-963d-42f5-adca-8862c7cbe16b-kube-api-access-755cq\") pod \"authorino-6f6944bcdd-hrl6h\" (UID: \"0d9e3e95-963d-42f5-adca-8862c7cbe16b\") " pod="kuadrant-system/authorino-6f6944bcdd-hrl6h"
Oct 31 19:34:43.076612 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.076569    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:43.162817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.162784    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hrl6h"
Oct 31 19:34:43.223907 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.223817    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hrl6h"]
Oct 31 19:34:43.249158 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.249110    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d46f6cbc6-4kfmc"]
Oct 31 19:34:43.258108 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.258075    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc"
Oct 31 19:34:43.292791 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.292736    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zgj5\" (UniqueName: \"kubernetes.io/projected/a1bf99ef-b919-48a0-abe6-1e63ff073265-kube-api-access-7zgj5\") pod \"authorino-6d46f6cbc6-4kfmc\" (UID: \"a1bf99ef-b919-48a0-abe6-1e63ff073265\") " pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc"
Oct 31 19:34:43.311674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.309782    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d46f6cbc6-4kfmc"]
Oct 31 19:34:43.393969 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.393926    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zgj5\" (UniqueName: \"kubernetes.io/projected/a1bf99ef-b919-48a0-abe6-1e63ff073265-kube-api-access-7zgj5\") pod \"authorino-6d46f6cbc6-4kfmc\" (UID: \"a1bf99ef-b919-48a0-abe6-1e63ff073265\") " pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc"
Oct 31 19:34:43.451961 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.451923    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zgj5\" (UniqueName: \"kubernetes.io/projected/a1bf99ef-b919-48a0-abe6-1e63ff073265-kube-api-access-7zgj5\") pod \"authorino-6d46f6cbc6-4kfmc\" (UID: \"a1bf99ef-b919-48a0-abe6-1e63ff073265\") " pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc"
Oct 31 19:34:43.552897 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:34:43.552865    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d9e3e95_963d_42f5_adca_8862c7cbe16b.slice/crio-8d3ff9f27042e868b36ba9875bad8f15d256a54ae924d29505ec88d84582bd21 WatchSource:0}: Error finding container 8d3ff9f27042e868b36ba9875bad8f15d256a54ae924d29505ec88d84582bd21: Status 404 returned error can't find the container with id 8d3ff9f27042e868b36ba9875bad8f15d256a54ae924d29505ec88d84582bd21
Oct 31 19:34:43.554254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.554225    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hrl6h"]
Oct 31 19:34:43.577583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.577523    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc"
Oct 31 19:34:43.702802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.702766    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d46f6cbc6-4kfmc"]
Oct 31 19:34:43.789253 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.789006    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hrl6h" event={"ID":"0d9e3e95-963d-42f5-adca-8862c7cbe16b","Type":"ContainerStarted","Data":"8d3ff9f27042e868b36ba9875bad8f15d256a54ae924d29505ec88d84582bd21"}
Oct 31 19:34:43.965290 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:34:43.965256    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1bf99ef_b919_48a0_abe6_1e63ff073265.slice/crio-2a20c0976ae6f00809f316999788efdb1aeb34562a7b52428df9e755356d9a54 WatchSource:0}: Error finding container 2a20c0976ae6f00809f316999788efdb1aeb34562a7b52428df9e755356d9a54: Status 404 returned error can't find the container with id 2a20c0976ae6f00809f316999788efdb1aeb34562a7b52428df9e755356d9a54
Oct 31 19:34:43.966996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:43.966971    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d46f6cbc6-4kfmc"]
Oct 31 19:34:44.795881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:44.795845    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc" event={"ID":"a1bf99ef-b919-48a0-abe6-1e63ff073265","Type":"ContainerStarted","Data":"2a20c0976ae6f00809f316999788efdb1aeb34562a7b52428df9e755356d9a54"}
Oct 31 19:34:45.084901 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:45.084811    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ks646 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:45.084901 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:45.084880    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:45.802950 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:45.802849    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc" event={"ID":"a1bf99ef-b919-48a0-abe6-1e63ff073265","Type":"ContainerStarted","Data":"8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b"}
Oct 31 19:34:45.802950 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:45.802890    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc" podUID="a1bf99ef-b919-48a0-abe6-1e63ff073265" containerName="authorino" containerID="cri-o://8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b" gracePeriod=30
Oct 31 19:34:45.804728 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:45.804696    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hrl6h" event={"ID":"0d9e3e95-963d-42f5-adca-8862c7cbe16b","Type":"ContainerStarted","Data":"6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd"}
Oct 31 19:34:45.804839 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:45.804725    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-hrl6h" podUID="0d9e3e95-963d-42f5-adca-8862c7cbe16b" containerName="authorino" containerID="cri-o://6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd" gracePeriod=30
Oct 31 19:34:45.920416 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:45.920358    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc" podStartSLOduration=1.5085256569999999 podStartE2EDuration="2.920339608s" totalImagesPullingTime="1.411813951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:45.919132921 +0000 UTC m=+1104.448666974" watchObservedRunningTime="2026-10-31 19:34:45.920339608 +0000 UTC m=+1104.449873664"
Oct 31 19:34:46.034055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.034006    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-hrl6h" podStartSLOduration=2.277622521 podStartE2EDuration="4.033991771s" totalImagesPullingTime="1.75636925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:46.033685737 +0000 UTC m=+1104.563219793" watchObservedRunningTime="2026-10-31 19:34:46.033991771 +0000 UTC m=+1104.563525824"
Oct 31 19:34:46.086862 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.086839    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc"
Oct 31 19:34:46.090028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.090007    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hrl6h"
Oct 31 19:34:46.139959 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.139927    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bf99ef-b919-48a0-abe6-1e63ff073265-kube-api-access-7zgj5\" (UniqueName: \"kubernetes.io/projected/a1bf99ef-b919-48a0-abe6-1e63ff073265-kube-api-access-7zgj5\") pod \"a1bf99ef-b919-48a0-abe6-1e63ff073265\" (UID: \"a1bf99ef-b919-48a0-abe6-1e63ff073265\") "
Oct 31 19:34:46.142263 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.142234    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bf99ef-b919-48a0-abe6-1e63ff073265-kube-api-access-7zgj5" pod "a1bf99ef-b919-48a0-abe6-1e63ff073265" (UID: "a1bf99ef-b919-48a0-abe6-1e63ff073265"). InnerVolumeSpecName "kube-api-access-7zgj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:34:46.241406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.241370    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9e3e95-963d-42f5-adca-8862c7cbe16b-kube-api-access-755cq\" (UniqueName: \"kubernetes.io/projected/0d9e3e95-963d-42f5-adca-8862c7cbe16b-kube-api-access-755cq\") pod \"0d9e3e95-963d-42f5-adca-8862c7cbe16b\" (UID: \"0d9e3e95-963d-42f5-adca-8862c7cbe16b\") "
Oct 31 19:34:46.241803 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.241529    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zgj5\" (UniqueName: \"kubernetes.io/projected/a1bf99ef-b919-48a0-abe6-1e63ff073265-kube-api-access-7zgj5\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:46.243553 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.243527    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9e3e95-963d-42f5-adca-8862c7cbe16b-kube-api-access-755cq" pod "0d9e3e95-963d-42f5-adca-8862c7cbe16b" (UID: "0d9e3e95-963d-42f5-adca-8862c7cbe16b"). InnerVolumeSpecName "kube-api-access-755cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:34:46.342294 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.342258    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-755cq\" (UniqueName: \"kubernetes.io/projected/0d9e3e95-963d-42f5-adca-8862c7cbe16b-kube-api-access-755cq\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:46.811099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.811066    5006 generic.go:356] "Generic (PLEG): container finished" podID="a1bf99ef-b919-48a0-abe6-1e63ff073265" containerID="8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b" exitCode=0
Oct 31 19:34:46.811581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.811110    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc"
Oct 31 19:34:46.811581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.811164    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc" event={"ID":"a1bf99ef-b919-48a0-abe6-1e63ff073265","Type":"ContainerDied","Data":"8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b"}
Oct 31 19:34:46.811581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.811219    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d46f6cbc6-4kfmc" event={"ID":"a1bf99ef-b919-48a0-abe6-1e63ff073265","Type":"ContainerDied","Data":"2a20c0976ae6f00809f316999788efdb1aeb34562a7b52428df9e755356d9a54"}
Oct 31 19:34:46.811581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.811247    5006 scope.go:122] "RemoveContainer" containerID="8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b"
Oct 31 19:34:46.812756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.812599    5006 generic.go:356] "Generic (PLEG): container finished" podID="0d9e3e95-963d-42f5-adca-8862c7cbe16b" containerID="6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd" exitCode=0
Oct 31 19:34:46.812756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.812638    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hrl6h"
Oct 31 19:34:46.812756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.812674    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hrl6h" event={"ID":"0d9e3e95-963d-42f5-adca-8862c7cbe16b","Type":"ContainerDied","Data":"6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd"}
Oct 31 19:34:46.812756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.812698    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hrl6h" event={"ID":"0d9e3e95-963d-42f5-adca-8862c7cbe16b","Type":"ContainerDied","Data":"8d3ff9f27042e868b36ba9875bad8f15d256a54ae924d29505ec88d84582bd21"}
Oct 31 19:34:46.822348 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.822283    5006 scope.go:122] "RemoveContainer" containerID="8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b"
Oct 31 19:34:46.822741 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:34:46.822709    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b\": container with ID starting with 8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b not found: ID does not exist" containerID="8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b"
Oct 31 19:34:46.822866 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.822751    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b"} err="failed to get container status \"8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b\": rpc error: code = NotFound desc = could not find container \"8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b\": container with ID starting with 8e30389cac7afa34428244c887aea05b6254dacf4bc5f8ab71cc5df590fda67b not found: ID does not exist"
Oct 31 19:34:46.822866 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.822776    5006 scope.go:122] "RemoveContainer" containerID="6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd"
Oct 31 19:34:46.833739 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.833712    5006 scope.go:122] "RemoveContainer" containerID="6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd"
Oct 31 19:34:46.834076 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:34:46.834060    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd\": container with ID starting with 6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd not found: ID does not exist" containerID="6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd"
Oct 31 19:34:46.834132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.834086    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd"} err="failed to get container status \"6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd\": rpc error: code = NotFound desc = could not find container \"6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd\": container with ID starting with 6e474c6ec106eaa6efb36ca49a27153e0d354cfc27574b532164b1304e5bd8cd not found: ID does not exist"
Oct 31 19:34:46.982632 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.982598    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hrl6h"]
Oct 31 19:34:46.986603 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:46.986577    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hrl6h"]
Oct 31 19:34:47.098138 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:47.098100    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d46f6cbc6-4kfmc"]
Oct 31 19:34:47.100994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:47.100971    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d46f6cbc6-4kfmc"]
Oct 31 19:34:47.597175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:47.597144    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d9e3e95-963d-42f5-adca-8862c7cbe16b" path="/var/lib/kubelet/pods/0d9e3e95-963d-42f5-adca-8862c7cbe16b/volumes"
Oct 31 19:34:47.597491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:47.597478    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bf99ef-b919-48a0-abe6-1e63ff073265" path="/var/lib/kubelet/pods/a1bf99ef-b919-48a0-abe6-1e63ff073265/volumes"
Oct 31 19:34:47.819724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:47.819689    5006 generic.go:356] "Generic (PLEG): container finished" podID="31b98296-1596-4837-8587-18e6ca94599b" containerID="4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288" exitCode=0
Oct 31 19:34:47.820168 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:47.819761    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" event={"ID":"31b98296-1596-4837-8587-18e6ca94599b","Type":"ContainerDied","Data":"4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288"}
Oct 31 19:34:47.820168 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:47.820135    5006 scope.go:122] "RemoveContainer" containerID="4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288"
Oct 31 19:34:47.821244 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:47.821211    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:34:47.821319 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:47.821293    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:34:47.821369 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:47.821328    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:34:48.102715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.102643    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"]
Oct 31 19:34:48.103306 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.103261    5006 cpu_manager.go:414] "containerMap: removing container" podUID="0d9e3e95-963d-42f5-adca-8862c7cbe16b" containerName="authorino"
Oct 31 19:34:48.103306 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.103285    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d9e3e95-963d-42f5-adca-8862c7cbe16b" containerName="authorino"
Oct 31 19:34:48.103387 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.103316    5006 cpu_manager.go:414] "containerMap: removing container" podUID="a1bf99ef-b919-48a0-abe6-1e63ff073265" containerName="authorino"
Oct 31 19:34:48.103387 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.103324    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bf99ef-b919-48a0-abe6-1e63ff073265" containerName="authorino"
Oct 31 19:34:48.103448 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.103408    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bf99ef-b919-48a0-abe6-1e63ff073265" containerName="authorino"
Oct 31 19:34:48.103448 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.103426    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d9e3e95-963d-42f5-adca-8862c7cbe16b" containerName="authorino"
Oct 31 19:34:48.106778 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.106758    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"
Oct 31 19:34:48.159166 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.159129    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"]
Oct 31 19:34:48.260071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.260036    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4xs\" (UniqueName: \"kubernetes.io/projected/c826795e-be35-4ba7-82c9-0d9e634d886d-kube-api-access-cw4xs\") pod \"llmisvc-controller-manager-6466c554d8-knn9s\" (UID: \"c826795e-be35-4ba7-82c9-0d9e634d886d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"
Oct 31 19:34:48.260242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.260094    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c826795e-be35-4ba7-82c9-0d9e634d886d-cert\") pod \"llmisvc-controller-manager-6466c554d8-knn9s\" (UID: \"c826795e-be35-4ba7-82c9-0d9e634d886d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"
Oct 31 19:34:48.361588 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.361474    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c826795e-be35-4ba7-82c9-0d9e634d886d-cert\") pod \"llmisvc-controller-manager-6466c554d8-knn9s\" (UID: \"c826795e-be35-4ba7-82c9-0d9e634d886d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"
Oct 31 19:34:48.361862 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.361598    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4xs\" (UniqueName: \"kubernetes.io/projected/c826795e-be35-4ba7-82c9-0d9e634d886d-kube-api-access-cw4xs\") pod \"llmisvc-controller-manager-6466c554d8-knn9s\" (UID: \"c826795e-be35-4ba7-82c9-0d9e634d886d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"
Oct 31 19:34:48.364133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.364109    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c826795e-be35-4ba7-82c9-0d9e634d886d-cert\") pod \"llmisvc-controller-manager-6466c554d8-knn9s\" (UID: \"c826795e-be35-4ba7-82c9-0d9e634d886d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"
Oct 31 19:34:48.423569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.423535    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4xs\" (UniqueName: \"kubernetes.io/projected/c826795e-be35-4ba7-82c9-0d9e634d886d-kube-api-access-cw4xs\") pod \"llmisvc-controller-manager-6466c554d8-knn9s\" (UID: \"c826795e-be35-4ba7-82c9-0d9e634d886d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"
Oct 31 19:34:48.720894 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.720800    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"
Oct 31 19:34:48.830398 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:48.830363    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" event={"ID":"31b98296-1596-4837-8587-18e6ca94599b","Type":"ContainerStarted","Data":"eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497"}
Oct 31 19:34:49.090110 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:34:49.090080    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc826795e_be35_4ba7_82c9_0d9e634d886d.slice/crio-771ead2632eab4070e240446147be7f1a4cba9a25548aec5303917dfebcae423 WatchSource:0}: Error finding container 771ead2632eab4070e240446147be7f1a4cba9a25548aec5303917dfebcae423: Status 404 returned error can't find the container with id 771ead2632eab4070e240446147be7f1a4cba9a25548aec5303917dfebcae423
Oct 31 19:34:49.091931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:49.091907    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"]
Oct 31 19:34:49.679528 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:49.679485    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:34:49.679808 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:49.679784    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ks646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Oct 31 19:34:49.679873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:49.679844    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Oct 31 19:34:49.836037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:49.835994    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s" event={"ID":"c826795e-be35-4ba7-82c9-0d9e634d886d","Type":"ContainerStarted","Data":"771ead2632eab4070e240446147be7f1a4cba9a25548aec5303917dfebcae423"}
Oct 31 19:34:50.680289 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:50.680254    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ks646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Oct 31 19:34:50.680518 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:50.680308    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Oct 31 19:34:50.844429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:50.844392    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s" event={"ID":"c826795e-be35-4ba7-82c9-0d9e634d886d","Type":"ContainerStarted","Data":"013ec9a6e19bb6bf42c1e07f134a3ce162da53b95e37beef04deeb5defc1b79e"}
Oct 31 19:34:50.844863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:50.844538    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"
Oct 31 19:34:50.986043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:50.985927    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s" podStartSLOduration=2.131369199 podStartE2EDuration="2.985912331s" totalImagesPullingTime="854.543132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:50.985286405 +0000 UTC m=+1109.514820459" watchObservedRunningTime="2026-10-31 19:34:50.985912331 +0000 UTC m=+1109.515446392"
Oct 31 19:34:51.680291 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:51.680253    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ks646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Oct 31 19:34:51.680491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:51.680305    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Oct 31 19:34:52.680444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:52.680408    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ks646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Oct 31 19:34:52.680889 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:52.680466    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Oct 31 19:34:53.680563 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:53.680531    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ks646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Oct 31 19:34:53.680999 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:53.680583    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Oct 31 19:34:54.683283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:54.683246    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:34:54.683700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:54.683637    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:34:54.684313 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:34:54.684295    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:35:05.392690 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:35:05.392590    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:35:17.855484 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:17.855451    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:21.852798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:21.852769    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-knn9s"
Oct 31 19:35:22.021099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.020804    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"]
Oct 31 19:35:22.021259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.021082    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp" podUID="a7c16946-8cf8-4974-912d-43a90d6371e0" containerName="manager" containerID="cri-o://a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794" gracePeriod=30
Oct 31 19:35:22.285440 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.285372    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:35:22.373322 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.373276    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c16946-8cf8-4974-912d-43a90d6371e0-kube-api-access-gc562\" (UniqueName: \"kubernetes.io/projected/a7c16946-8cf8-4974-912d-43a90d6371e0-kube-api-access-gc562\") pod \"a7c16946-8cf8-4974-912d-43a90d6371e0\" (UID: \"a7c16946-8cf8-4974-912d-43a90d6371e0\") "
Oct 31 19:35:22.373534 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.373376    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c16946-8cf8-4974-912d-43a90d6371e0-cert\" (UniqueName: \"kubernetes.io/secret/a7c16946-8cf8-4974-912d-43a90d6371e0-cert\") pod \"a7c16946-8cf8-4974-912d-43a90d6371e0\" (UID: \"a7c16946-8cf8-4974-912d-43a90d6371e0\") "
Oct 31 19:35:22.375524 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.375492    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c16946-8cf8-4974-912d-43a90d6371e0-kube-api-access-gc562" pod "a7c16946-8cf8-4974-912d-43a90d6371e0" (UID: "a7c16946-8cf8-4974-912d-43a90d6371e0"). InnerVolumeSpecName "kube-api-access-gc562". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:22.375643 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.375524    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c16946-8cf8-4974-912d-43a90d6371e0-cert" pod "a7c16946-8cf8-4974-912d-43a90d6371e0" (UID: "a7c16946-8cf8-4974-912d-43a90d6371e0"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:35:22.474239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.474200    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc562\" (UniqueName: \"kubernetes.io/projected/a7c16946-8cf8-4974-912d-43a90d6371e0-kube-api-access-gc562\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:22.474239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.474233    5006 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7c16946-8cf8-4974-912d-43a90d6371e0-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:22.811498 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.811463    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:22.998012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.997971    5006 generic.go:356] "Generic (PLEG): container finished" podID="a7c16946-8cf8-4974-912d-43a90d6371e0" containerID="a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794" exitCode=0
Oct 31 19:35:22.998012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.998016    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"
Oct 31 19:35:22.998563 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.998047    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp" event={"ID":"a7c16946-8cf8-4974-912d-43a90d6371e0","Type":"ContainerDied","Data":"a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794"}
Oct 31 19:35:22.998563 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.998097    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp" event={"ID":"a7c16946-8cf8-4974-912d-43a90d6371e0","Type":"ContainerDied","Data":"9d7ce86af6e44ff37c264928179896690493e5fce5bfc5adbdef704253717ab1"}
Oct 31 19:35:22.998563 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:22.998119    5006 scope.go:122] "RemoveContainer" containerID="a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794"
Oct 31 19:35:23.006917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:23.006896    5006 scope.go:122] "RemoveContainer" containerID="a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794"
Oct 31 19:35:23.007190 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:35:23.007172    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794\": container with ID starting with a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794 not found: ID does not exist" containerID="a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794"
Oct 31 19:35:23.007265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:23.007199    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794"} err="failed to get container status \"a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794\": rpc error: code = NotFound desc = could not find container \"a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794\": container with ID starting with a1762c5d6d0dfb930153f788c486ab621e5b52f70c1813ff0dbf871f7829d794 not found: ID does not exist"
Oct 31 19:35:23.040874 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:23.040841    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:23.165060 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:23.165020    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"]
Oct 31 19:35:23.215447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:23.215414    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g85kp"]
Oct 31 19:35:23.596477 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:23.596442    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c16946-8cf8-4974-912d-43a90d6371e0" path="/var/lib/kubelet/pods/a7c16946-8cf8-4974-912d-43a90d6371e0/volumes"
Oct 31 19:35:35.393024 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:35:35.392983    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:35:42.592518 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.592475    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"]
Oct 31 19:35:42.593101 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.593080    5006 cpu_manager.go:414] "containerMap: removing container" podUID="a7c16946-8cf8-4974-912d-43a90d6371e0" containerName="manager"
Oct 31 19:35:42.593180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.593103    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c16946-8cf8-4974-912d-43a90d6371e0" containerName="manager"
Oct 31 19:35:42.593246 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.593232    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c16946-8cf8-4974-912d-43a90d6371e0" containerName="manager"
Oct 31 19:35:42.596443 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.596419    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.669898 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.669856    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.670085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.669927    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.670085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.669977    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wd2\" (UniqueName: \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-kube-api-access-47wd2\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.670085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.670061    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.670210 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.670085    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.670210 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.670107    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683ca9eb-8f9e-44db-b22b-15cc698948d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.670210 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.670158    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.670210 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.670181    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.670341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.670223    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.713564 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.713517    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"]
Oct 31 19:35:42.771048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771008    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771048    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wd2\" (UniqueName: \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-kube-api-access-47wd2\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771067    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771082    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771218    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683ca9eb-8f9e-44db-b22b-15cc698948d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771261    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771291    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771328    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771436    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771507    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771564    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771766 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771744    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.771993 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.771962    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683ca9eb-8f9e-44db-b22b-15cc698948d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.772099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.772021    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.773711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.773681    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.773711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.773693    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.828723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.828687    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.828893 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.828744    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wd2\" (UniqueName: \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-kube-api-access-47wd2\") pod \"maas-default-gateway-openshift-default-58d898bd55-6jhdc\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:42.909413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:42.909325    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:43.208217 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:43.208121    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:43.326108 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:35:43.326076    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683ca9eb_8f9e_44db_b22b_15cc698948d2.slice/crio-612edd9e623744fb3e3598332f026af925e12994f70b1faef3419a6683fb4cfe WatchSource:0}: Error finding container 612edd9e623744fb3e3598332f026af925e12994f70b1faef3419a6683fb4cfe: Status 404 returned error can't find the container with id 612edd9e623744fb3e3598332f026af925e12994f70b1faef3419a6683fb4cfe
Oct 31 19:35:43.327366 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:43.327346    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"]
Oct 31 19:35:43.328361 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:43.328325    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:43.328442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:43.328405    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:43.328476 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:43.328449    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:44.112755 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:44.111768    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" event={"ID":"683ca9eb-8f9e-44db-b22b-15cc698948d2","Type":"ContainerStarted","Data":"6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d"}
Oct 31 19:35:44.112755 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:44.111849    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" event={"ID":"683ca9eb-8f9e-44db-b22b-15cc698948d2","Type":"ContainerStarted","Data":"612edd9e623744fb3e3598332f026af925e12994f70b1faef3419a6683fb4cfe"}
Oct 31 19:35:44.289357 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:44.289124    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" podStartSLOduration=2.289108613 podStartE2EDuration="2.289108613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:44.287992797 +0000 UTC m=+1162.817526850" watchObservedRunningTime="2026-10-31 19:35:44.289108613 +0000 UTC m=+1162.818642665"
Oct 31 19:35:44.915080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:44.915040    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:44.917467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:44.917432    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6jhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:35:44.917716 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:44.917684    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 19:35:45.910819 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:45.910782    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6jhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:35:45.911415 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:45.911378    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 19:35:46.544668 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:46.544618    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:35:46.910248 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:46.910206    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6jhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:35:46.910597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:46.910549    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 19:35:47.914138 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:47.914095    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6jhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:35:47.914825 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:47.914789    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 19:35:48.913675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:48.913099    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6jhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:35:48.913675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:48.913181    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 19:35:49.913769 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:49.913736    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:49.914236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:49.914027    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:49.914962 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:49.914944    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:35:52.279597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:52.279564    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:54.147172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:54.147132    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:35:55.708867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:55.708833    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:57.606213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.606178    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:57.709522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.709489    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"]
Oct 31 19:35:57.714457 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.714430    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.733050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.733014    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.733229 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.733077    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.733327 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.733298    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.733392 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.733346    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fqt8\" (UniqueName: \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-kube-api-access-2fqt8\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.733392 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.733381    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.733499 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.733446    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.733499 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.733477    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.733612 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.733508    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.733612 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.733586    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.821868 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.821830    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"]
Oct 31 19:35:57.834763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.834725    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.834763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.834765    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fqt8\" (UniqueName: \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-kube-api-access-2fqt8\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.834788    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.834815    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.834834    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.834855    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.834884    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.834921    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.834974    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.835255    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.835399    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.835431    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835554 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.835479    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.835863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.835839    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.837236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.837205    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.837346 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.837329    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.892010 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.891933    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:57.892149 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:57.892112    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fqt8\" (UniqueName: \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-kube-api-access-2fqt8\") pod \"maas-default-gateway-openshift-default-58d898bd55-vxw57\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:58.034264 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:58.034220    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:35:58.450595 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:35:58.450566    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fa87d5_f9ce_4a19_a70f_fea496c8c40d.slice/crio-b151c4f3c4c9b7b4c8be4db096ee47ce451712f70683aab42e20ae45e92c2b72 WatchSource:0}: Error finding container b151c4f3c4c9b7b4c8be4db096ee47ce451712f70683aab42e20ae45e92c2b72: Status 404 returned error can't find the container with id b151c4f3c4c9b7b4c8be4db096ee47ce451712f70683aab42e20ae45e92c2b72
Oct 31 19:35:58.453333 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:58.453302    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"]
Oct 31 19:35:58.453599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:58.453561    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:58.453815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:58.453772    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:58.453934 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:58.453838    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:59.223243 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:59.223199    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" event={"ID":"00fa87d5-f9ce-4a19-a70f-fea496c8c40d","Type":"ContainerStarted","Data":"98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f"}
Oct 31 19:35:59.223688 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:59.223271    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" event={"ID":"00fa87d5-f9ce-4a19-a70f-fea496c8c40d","Type":"ContainerStarted","Data":"b151c4f3c4c9b7b4c8be4db096ee47ce451712f70683aab42e20ae45e92c2b72"}
Oct 31 19:35:59.342404 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:35:59.342352    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" podStartSLOduration=2.342337637 podStartE2EDuration="2.342337637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:59.341566632 +0000 UTC m=+1177.871100683" watchObservedRunningTime="2026-10-31 19:35:59.342337637 +0000 UTC m=+1177.871871689"
Oct 31 19:36:00.035030 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:00.034984    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:36:00.036386 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:00.036359    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vxw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 19:36:00.036523 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:00.036418    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 19:36:01.035249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:01.035211    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vxw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 19:36:01.035683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:01.035281    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 19:36:01.265385 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:01.265307    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:36:02.019491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:02.019443    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:02.044639 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:02.044602    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vxw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 19:36:02.046808 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:02.046776    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 19:36:03.035434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:03.035395    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vxw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 19:36:03.035689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:03.035460    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 19:36:04.035400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:04.035359    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vxw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 19:36:04.035955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:04.035435    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 19:36:05.114528 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:05.114482    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:05.393782 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:36:05.393690    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:36:06.032668 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:06.032616    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:36:06.033059 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:06.033036    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:36:06.034043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:06.034020    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:36:10.541230 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:10.541193    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:14.118887 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:14.118850    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:36:14.707974 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:14.707936    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:18.181692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:18.181641    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:19.462867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:19.462832    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:21.734679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:21.734619    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:36:21.786671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:21.783874    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:36:26.168640 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:26.168604    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:36:29.129226 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:29.129187    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:35.394519 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:36:35.394437    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:36:42.923639 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:42.923604    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ccc6db5f7-5g26l"]
Oct 31 19:36:42.926978 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:42.926952    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccc6db5f7-5g26l"
Oct 31 19:36:42.980345 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:42.980313    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ccc6db5f7-5g26l"]
Oct 31 19:36:42.995378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:42.995342    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbkcj\" (UniqueName: \"kubernetes.io/projected/86cd364e-be8d-48bb-a86d-fb588b2fd6c9-kube-api-access-sbkcj\") pod \"authorino-5ccc6db5f7-5g26l\" (UID: \"86cd364e-be8d-48bb-a86d-fb588b2fd6c9\") " pod="kuadrant-system/authorino-5ccc6db5f7-5g26l"
Oct 31 19:36:42.995543 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:42.995459    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/86cd364e-be8d-48bb-a86d-fb588b2fd6c9-tls-cert\") pod \"authorino-5ccc6db5f7-5g26l\" (UID: \"86cd364e-be8d-48bb-a86d-fb588b2fd6c9\") " pod="kuadrant-system/authorino-5ccc6db5f7-5g26l"
Oct 31 19:36:43.028925 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:43.028892    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 19:36:43.030441 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:43.030406    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ldbjt\""
Oct 31 19:36:43.096091 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:43.096055    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbkcj\" (UniqueName: \"kubernetes.io/projected/86cd364e-be8d-48bb-a86d-fb588b2fd6c9-kube-api-access-sbkcj\") pod \"authorino-5ccc6db5f7-5g26l\" (UID: \"86cd364e-be8d-48bb-a86d-fb588b2fd6c9\") " pod="kuadrant-system/authorino-5ccc6db5f7-5g26l"
Oct 31 19:36:43.096292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:43.096134    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/86cd364e-be8d-48bb-a86d-fb588b2fd6c9-tls-cert\") pod \"authorino-5ccc6db5f7-5g26l\" (UID: \"86cd364e-be8d-48bb-a86d-fb588b2fd6c9\") " pod="kuadrant-system/authorino-5ccc6db5f7-5g26l"
Oct 31 19:36:43.098586 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:43.098554    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/86cd364e-be8d-48bb-a86d-fb588b2fd6c9-tls-cert\") pod \"authorino-5ccc6db5f7-5g26l\" (UID: \"86cd364e-be8d-48bb-a86d-fb588b2fd6c9\") " pod="kuadrant-system/authorino-5ccc6db5f7-5g26l"
Oct 31 19:36:43.156416 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:43.156387    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbkcj\" (UniqueName: \"kubernetes.io/projected/86cd364e-be8d-48bb-a86d-fb588b2fd6c9-kube-api-access-sbkcj\") pod \"authorino-5ccc6db5f7-5g26l\" (UID: \"86cd364e-be8d-48bb-a86d-fb588b2fd6c9\") " pod="kuadrant-system/authorino-5ccc6db5f7-5g26l"
Oct 31 19:36:43.162088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:43.162040    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:43.238827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:43.238743    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccc6db5f7-5g26l"
Oct 31 19:36:43.459957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:43.459922    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:43.609457 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:36:43.609421    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86cd364e_be8d_48bb_a86d_fb588b2fd6c9.slice/crio-d92330743a343f402b073657673c8fae5c6cdaa8d153842eee07780dd9ff86ac WatchSource:0}: Error finding container d92330743a343f402b073657673c8fae5c6cdaa8d153842eee07780dd9ff86ac: Status 404 returned error can't find the container with id d92330743a343f402b073657673c8fae5c6cdaa8d153842eee07780dd9ff86ac
Oct 31 19:36:43.611475 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:43.611443    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ccc6db5f7-5g26l"]
Oct 31 19:36:44.510942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:44.510907    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ccc6db5f7-5g26l" event={"ID":"86cd364e-be8d-48bb-a86d-fb588b2fd6c9","Type":"ContainerStarted","Data":"d92330743a343f402b073657673c8fae5c6cdaa8d153842eee07780dd9ff86ac"}
Oct 31 19:36:45.517274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:45.517181    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ccc6db5f7-5g26l" event={"ID":"86cd364e-be8d-48bb-a86d-fb588b2fd6c9","Type":"ContainerStarted","Data":"0d2f8eefbe0014f40450149e55d5f03aff1e8c9019c8db1332ca2cc77d363843"}
Oct 31 19:36:45.635049 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:45.634992    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5ccc6db5f7-5g26l" podStartSLOduration=1.9965161729999998 podStartE2EDuration="3.634977174s" totalImagesPullingTime="1.638461001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:45.633745593 +0000 UTC m=+1224.163279647" watchObservedRunningTime="2026-10-31 19:36:45.634977174 +0000 UTC m=+1224.164511226"
Oct 31 19:36:50.568512 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:50.568422    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:50.834221 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:36:50.834133    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:05.393986 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:37:05.393949    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:37:14.925238 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:14.925198    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:15.383866 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:15.383829    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:16.946890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:16.946853    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:17.378933 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:17.378896    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:18.532937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:18.532900    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:18.777512 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:18.777479    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:35.393913 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:37:35.393875    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:37:42.987242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:42.987207    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:43.220515 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:43.220473    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:43.677239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:43.677204    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:43.902644 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:43.902608    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:44.133291 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:44.133260    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:44.360173 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:44.360139    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:44.584554 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:44.584463    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:44.809988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:44.809948    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:45.043406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:45.043358    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:45.279425 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:45.279386    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:45.508093 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:45.508058    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:45.736410 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:45.736375    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:48.469682 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:48.469632    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:48.691882 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:48.691833    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:49.482014 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:49.481984    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:49.709119 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:49.709090    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:50.125865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:50.125833    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:50.372956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:50.372917    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:50.730541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:50.730510    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:50.964034 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:50.964003    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:51.375038 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:51.375006    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:51.604353 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:51.604308    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:53.085151 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:53.085116    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:53.336850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:53.336820    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:53.571527 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:53.571491    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:53.799135 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:53.799058    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:54.029360 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:54.029319    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:58.595527 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:58.595482    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:58.822718 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:37:58.822684    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:00.629412 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:00.629372    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:00.846308 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:00.846237    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:01.203063 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.203025    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:01.428758 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.428722    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:01.677725 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.677689    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"]
Oct 31 19:38:01.681756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.681736    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 19:38:01.735692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.735640    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"]
Oct 31 19:38:01.747314 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.747282    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96b94c3-1d84-46a1-8898-d9a32e19816f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9\" (UID: \"c96b94c3-1d84-46a1-8898-d9a32e19816f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 19:38:01.747523 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.747412    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdzhb\" (UniqueName: \"kubernetes.io/projected/c96b94c3-1d84-46a1-8898-d9a32e19816f-kube-api-access-fdzhb\") pod \"payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9\" (UID: \"c96b94c3-1d84-46a1-8898-d9a32e19816f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 19:38:01.783818 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.783783    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-570915\""
Oct 31 19:38:01.783988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.783911    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-570915-dockercfg-z7ctv\""
Oct 31 19:38:01.848023 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.847977    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96b94c3-1d84-46a1-8898-d9a32e19816f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9\" (UID: \"c96b94c3-1d84-46a1-8898-d9a32e19816f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 19:38:01.848209 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.848081    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzhb\" (UniqueName: \"kubernetes.io/projected/c96b94c3-1d84-46a1-8898-d9a32e19816f-kube-api-access-fdzhb\") pod \"payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9\" (UID: \"c96b94c3-1d84-46a1-8898-d9a32e19816f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 19:38:01.848795 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.848774    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96b94c3-1d84-46a1-8898-d9a32e19816f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9\" (UID: \"c96b94c3-1d84-46a1-8898-d9a32e19816f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 19:38:01.906253 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.906221    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzhb\" (UniqueName: \"kubernetes.io/projected/c96b94c3-1d84-46a1-8898-d9a32e19816f-kube-api-access-fdzhb\") pod \"payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9\" (UID: \"c96b94c3-1d84-46a1-8898-d9a32e19816f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 19:38:01.994894 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:01.994803    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 19:38:02.368544 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:38:02.368505    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96b94c3_1d84_46a1_8898_d9a32e19816f.slice/crio-2d760cb8ff652c6185e24e2df6fe97795105b0183d74b148f8a117a2874a00a1 WatchSource:0}: Error finding container 2d760cb8ff652c6185e24e2df6fe97795105b0183d74b148f8a117a2874a00a1: Status 404 returned error can't find the container with id 2d760cb8ff652c6185e24e2df6fe97795105b0183d74b148f8a117a2874a00a1
Oct 31 19:38:02.368736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:02.368714    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"]
Oct 31 19:38:02.904321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:02.904279    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9" event={"ID":"c96b94c3-1d84-46a1-8898-d9a32e19816f","Type":"ContainerStarted","Data":"2d760cb8ff652c6185e24e2df6fe97795105b0183d74b148f8a117a2874a00a1"}
Oct 31 19:38:03.721799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:03.721760    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:03.944448 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:03.944413    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:04.043152 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.043072    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"]
Oct 31 19:38:04.046358 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.046335    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 19:38:04.063276 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.063243    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ftcb\" (UniqueName: \"kubernetes.io/projected/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-kube-api-access-2ftcb\") pod \"payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m\" (UID: \"432c71ed-404c-4b6d-a521-0ed8ff30f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 19:38:04.063432 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.063294    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m\" (UID: \"432c71ed-404c-4b6d-a521-0ed8ff30f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 19:38:04.101544 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.101498    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"]
Oct 31 19:38:04.150107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.150074    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-7cfafd-dockercfg-cw5cz\""
Oct 31 19:38:04.150302 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.150077    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-7cfafd\""
Oct 31 19:38:04.163852 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.163808    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m\" (UID: \"432c71ed-404c-4b6d-a521-0ed8ff30f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 19:38:04.164024 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.163958    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ftcb\" (UniqueName: \"kubernetes.io/projected/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-kube-api-access-2ftcb\") pod \"payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m\" (UID: \"432c71ed-404c-4b6d-a521-0ed8ff30f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 19:38:04.164535 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.164512    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m\" (UID: \"432c71ed-404c-4b6d-a521-0ed8ff30f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 19:38:04.224748 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.224718    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ftcb\" (UniqueName: \"kubernetes.io/projected/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-kube-api-access-2ftcb\") pod \"payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m\" (UID: \"432c71ed-404c-4b6d-a521-0ed8ff30f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 19:38:04.365902 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.365865    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 19:38:04.736470 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:38:04.736426    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432c71ed_404c_4b6d_a521_0ed8ff30f8ea.slice/crio-98c78cd75faf30cce2418c9c77161ac5efc5095b7de3ae6a1ce0aac83e9c0be9 WatchSource:0}: Error finding container 98c78cd75faf30cce2418c9c77161ac5efc5095b7de3ae6a1ce0aac83e9c0be9: Status 404 returned error can't find the container with id 98c78cd75faf30cce2418c9c77161ac5efc5095b7de3ae6a1ce0aac83e9c0be9
Oct 31 19:38:04.738500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.738469    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"]
Oct 31 19:38:04.921091 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:04.921053    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m" event={"ID":"432c71ed-404c-4b6d-a521-0ed8ff30f8ea","Type":"ContainerStarted","Data":"98c78cd75faf30cce2418c9c77161ac5efc5095b7de3ae6a1ce0aac83e9c0be9"}
Oct 31 19:38:05.400231 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:38:05.400186    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:38:05.813739 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:05.813705    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:05.927389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:05.927352    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9" event={"ID":"c96b94c3-1d84-46a1-8898-d9a32e19816f","Type":"ContainerStarted","Data":"a991994978f70d09c077c14a4ce7513f8b6dcd506d32a9c13ffd52ad2d256c8e"}
Oct 31 19:38:05.927540 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:05.927452    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 19:38:06.035148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:06.035111    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:06.044389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:06.044331    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9" podStartSLOduration=1.612492315 podStartE2EDuration="5.044316176s" totalImagesPullingTime="3.431823861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:06.043132437 +0000 UTC m=+1304.572666521" watchObservedRunningTime="2026-10-31 19:38:06.044316176 +0000 UTC m=+1304.573850227"
Oct 31 19:38:06.934534 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:06.934491    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m" event={"ID":"432c71ed-404c-4b6d-a521-0ed8ff30f8ea","Type":"ContainerStarted","Data":"8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd"}
Oct 31 19:38:07.050372 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:07.050315    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m" podStartSLOduration=2.643571091 podStartE2EDuration="4.050296572s" totalImagesPullingTime="1.406725481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:07.048356848 +0000 UTC m=+1305.577890920" watchObservedRunningTime="2026-10-31 19:38:07.050296572 +0000 UTC m=+1305.579830626"
Oct 31 19:38:07.939176 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:07.939147    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 19:38:08.222390 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:08.222290    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:38:10.887088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:10.887046    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:11.126444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.126410    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:11.213720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.213623    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"]
Oct 31 19:38:11.217424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.217406    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 19:38:11.236666 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.236620    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn\" (UID: \"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 19:38:11.236817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.236744    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrcz5\" (UniqueName: \"kubernetes.io/projected/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-kube-api-access-mrcz5\") pod \"payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn\" (UID: \"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 19:38:11.271757 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.271694    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"]
Oct 31 19:38:11.319667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.319614    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-662be1\""
Oct 31 19:38:11.319873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.319688    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-662be1-dockercfg-pc8tc\""
Oct 31 19:38:11.338130 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.338093    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn\" (UID: \"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 19:38:11.338343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.338151    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrcz5\" (UniqueName: \"kubernetes.io/projected/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-kube-api-access-mrcz5\") pod \"payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn\" (UID: \"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 19:38:11.338975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.338952    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn\" (UID: \"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 19:38:11.363463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.363428    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:11.371970 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.371936    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"]
Oct 31 19:38:11.375913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.375893    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 19:38:11.395114 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.395083    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrcz5\" (UniqueName: \"kubernetes.io/projected/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-kube-api-access-mrcz5\") pod \"payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn\" (UID: \"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 19:38:11.422389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.422355    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"]
Oct 31 19:38:11.438934 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.438894    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9\" (UID: \"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 19:38:11.439172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.438979    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9vhl\" (UniqueName: \"kubernetes.io/projected/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-kube-api-access-t9vhl\") pod \"payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9\" (UID: \"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 19:38:11.530568 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.530473    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 19:38:11.539981 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.539951    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9vhl\" (UniqueName: \"kubernetes.io/projected/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-kube-api-access-t9vhl\") pod \"payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9\" (UID: \"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 19:38:11.540107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.540061    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9\" (UID: \"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 19:38:11.540770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.540751    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9\" (UID: \"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 19:38:11.605286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.605252    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9vhl\" (UniqueName: \"kubernetes.io/projected/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-kube-api-access-t9vhl\") pod \"payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9\" (UID: \"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 19:38:11.689312 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.689261    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 19:38:11.896695 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:38:11.896644    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ad6ab8_45f3_4ae4_88f8_7572147cabb7.slice/crio-2dfd6606ef3b79ba9d061397582be975a7fe8e96e8c984b21baf7320f1f8f219 WatchSource:0}: Error finding container 2dfd6606ef3b79ba9d061397582be975a7fe8e96e8c984b21baf7320f1f8f219: Status 404 returned error can't find the container with id 2dfd6606ef3b79ba9d061397582be975a7fe8e96e8c984b21baf7320f1f8f219
Oct 31 19:38:11.898805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.898779    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"]
Oct 31 19:38:11.964919 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.964882    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn" event={"ID":"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7","Type":"ContainerStarted","Data":"81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d"}
Oct 31 19:38:11.965107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.964940    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn" event={"ID":"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7","Type":"ContainerStarted","Data":"2dfd6606ef3b79ba9d061397582be975a7fe8e96e8c984b21baf7320f1f8f219"}
Oct 31 19:38:11.965107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:11.965058    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 19:38:12.055575 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:38:12.055545    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d0fa26_cffd_44a1_b12a_ba9d6207d3d7.slice/crio-0627cc5259e38e55392aed17ea948301f146c3bb279e6de44a39e426a662a22a WatchSource:0}: Error finding container 0627cc5259e38e55392aed17ea948301f146c3bb279e6de44a39e426a662a22a: Status 404 returned error can't find the container with id 0627cc5259e38e55392aed17ea948301f146c3bb279e6de44a39e426a662a22a
Oct 31 19:38:12.058140 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:12.058116    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"]
Oct 31 19:38:12.080454 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:12.080397    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn" podStartSLOduration=1.080377884 podStartE2EDuration="1.080377884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:12.079321172 +0000 UTC m=+1310.608855227" watchObservedRunningTime="2026-10-31 19:38:12.080377884 +0000 UTC m=+1310.609911940"
Oct 31 19:38:12.982258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:12.982212    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9" event={"ID":"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7","Type":"ContainerStarted","Data":"d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46"}
Oct 31 19:38:12.982774 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:12.982316    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9" event={"ID":"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7","Type":"ContainerStarted","Data":"0627cc5259e38e55392aed17ea948301f146c3bb279e6de44a39e426a662a22a"}
Oct 31 19:38:12.982774 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:12.982518    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 19:38:13.099175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:13.099115    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9" podStartSLOduration=2.099103161 podStartE2EDuration="2.099103161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:13.098688048 +0000 UTC m=+1311.628222098" watchObservedRunningTime="2026-10-31 19:38:13.099103161 +0000 UTC m=+1311.628637213"
Oct 31 19:38:15.792139 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:15.792101    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:16.040637 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:16.040599    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:16.264832 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:16.264797    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:16.480080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:16.480030    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:16.939195 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:16.939164    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 19:38:17.048229 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:17.048191    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:38:18.309201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.309146    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:18.534174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.534134    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:18.637540 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.637508    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"]
Oct 31 19:38:18.641748 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.641724    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 19:38:18.696576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.696543    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"]
Oct 31 19:38:18.709217 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.709184    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd68c\" (UniqueName: \"kubernetes.io/projected/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-kube-api-access-kd68c\") pod \"payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk\" (UID: \"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 19:38:18.709464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.709440    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk\" (UID: \"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 19:38:18.760316 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.760134    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-967dee-dockercfg-qxcx5\""
Oct 31 19:38:18.760316 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.760274    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-967dee\""
Oct 31 19:38:18.811141 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.811097    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd68c\" (UniqueName: \"kubernetes.io/projected/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-kube-api-access-kd68c\") pod \"payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk\" (UID: \"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 19:38:18.811334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.811257    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk\" (UID: \"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 19:38:18.811972 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.811953    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk\" (UID: \"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 19:38:18.861970 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.861930    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:18.868562 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.868532    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd68c\" (UniqueName: \"kubernetes.io/projected/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-kube-api-access-kd68c\") pod \"payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk\" (UID: \"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 19:38:18.949945 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.949645    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 19:38:18.957735 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:18.957704    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 19:38:19.344192 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:38:19.344153    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e920c5_f1ee_4aaa_ae85_88fa1e4aea58.slice/crio-40f896458fd627ae01598ec4910e2ec727b1e3f2412c91b2072b766b12b6ccab WatchSource:0}: Error finding container 40f896458fd627ae01598ec4910e2ec727b1e3f2412c91b2072b766b12b6ccab: Status 404 returned error can't find the container with id 40f896458fd627ae01598ec4910e2ec727b1e3f2412c91b2072b766b12b6ccab
Oct 31 19:38:19.348461 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:19.348440    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"]
Oct 31 19:38:20.020972 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:20.020933    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk" event={"ID":"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58","Type":"ContainerStarted","Data":"99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387"}
Oct 31 19:38:20.021143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:20.020992    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk" event={"ID":"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58","Type":"ContainerStarted","Data":"40f896458fd627ae01598ec4910e2ec727b1e3f2412c91b2072b766b12b6ccab"}
Oct 31 19:38:20.021143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:20.021035    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 19:38:20.137267 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:20.137204    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk" podStartSLOduration=2.137187698 podStartE2EDuration="2.137187698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:20.136484296 +0000 UTC m=+1318.666018346" watchObservedRunningTime="2026-10-31 19:38:20.137187698 +0000 UTC m=+1318.666721751"
Oct 31 19:38:21.455869 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:21.455831    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:22.985887 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:22.985855    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 19:38:23.991074 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:23.991045    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 19:38:24.517450 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:24.517409    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:38:26.443698 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:26.443636    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:26.666212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:26.666181    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:31.029926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:31.029893    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 19:38:31.600888 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:31.600847    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:31.827310 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:31.827272    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:34.654935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:34.654891    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:38:35.394669 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:38:35.394614    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:38:36.783110 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:36.783078    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:41.628194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:41.628156    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:41.842527 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:41.842493    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:43.635075 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:43.635026    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:38:44.994181 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:44.994148    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:45.618537 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:45.618503    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:45.876908 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:45.876820    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:47.113350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:47.113315    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:48.754495 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.754458    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"]
Oct 31 19:38:48.760777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.760752    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.812179 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.812145    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"]
Oct 31 19:38:48.829660 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.829611    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tls-certs\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.829833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.829783    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-dshm\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.829833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.829805    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-home\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.829833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.829825    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-model-cache\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.829954 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.829904    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86gk\" (UniqueName: \"kubernetes.io/projected/8d8f7246-2e48-4496-9934-c5eb5d4b3134-kube-api-access-w86gk\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.830005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.829971    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tmp-dir\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.860242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.860201    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"openshift-service-ca.crt\""
Oct 31 19:38:48.860242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.860233    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"simulator-w3-662be1-kserve-self-signed-certs\""
Oct 31 19:38:48.863732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.863704    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"kube-root-ca.crt\""
Oct 31 19:38:48.863732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.863705    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"default-dockercfg-4cd7k\""
Oct 31 19:38:48.931233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.931190    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-dshm\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.931233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.931231    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-home\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.931502 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.931252    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-model-cache\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.931502 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.931321    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86gk\" (UniqueName: \"kubernetes.io/projected/8d8f7246-2e48-4496-9934-c5eb5d4b3134-kube-api-access-w86gk\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.931502 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.931395    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tmp-dir\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.931502 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.931437    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tls-certs\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.931764 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.931609    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-model-cache\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.931830 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.931793    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tmp-dir\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.931907 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.931884    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-home\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.933424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.933396    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:48.933613 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.933592    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-dshm\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.934127 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.934106    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tls-certs\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:48.989177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:48.989147    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86gk\" (UniqueName: \"kubernetes.io/projected/8d8f7246-2e48-4496-9934-c5eb5d4b3134-kube-api-access-w86gk\") pod \"simulator-w3-662be1-kserve-5b6688c44d-5cxs8\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") " pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:49.073001 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:49.072912    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:49.449137 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:38:49.449098    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d8f7246_2e48_4496_9934_c5eb5d4b3134.slice/crio-e8017663889458ba67e662b295fd4e448b9971714c688ce2cc043df83ba4d8fe WatchSource:0}: Error finding container e8017663889458ba67e662b295fd4e448b9971714c688ce2cc043df83ba4d8fe: Status 404 returned error can't find the container with id e8017663889458ba67e662b295fd4e448b9971714c688ce2cc043df83ba4d8fe
Oct 31 19:38:49.451347 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:49.451330    5006 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:38:49.451562 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:49.451542    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"]
Oct 31 19:38:50.181909 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:50.181858    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" event={"ID":"8d8f7246-2e48-4496-9934-c5eb5d4b3134","Type":"ContainerStarted","Data":"e8017663889458ba67e662b295fd4e448b9971714c688ce2cc043df83ba4d8fe"}
Oct 31 19:38:51.190955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:51.190904    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" event={"ID":"8d8f7246-2e48-4496-9934-c5eb5d4b3134","Type":"ContainerStarted","Data":"5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc"}
Oct 31 19:38:51.324408 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:51.324291    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podStartSLOduration=1.787659352 podStartE2EDuration="3.324273182s" totalImagesPullingTime="1.53661383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:51.321100312 +0000 UTC m=+1349.850634364" watchObservedRunningTime="2026-10-31 19:38:51.324273182 +0000 UTC m=+1349.853807237"
Oct 31 19:38:52.073023 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:52.072990    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:52.176566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:52.176531    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:54.375552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:54.375519    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:54.931561 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:54.931532    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:55.158500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:55.158465    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:57.642596 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:57.642565    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:58.834682 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:58.834633    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:38:59.073429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:59.073351    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:59.086450 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:59.086422    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:38:59.251328 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:38:59.251292    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:39:00.761721 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:00.761684    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:01.420834 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:01.420797    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:01.708348 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:01.708259    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:03.503584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:03.503549    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:04.608030 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:04.607999    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:05.230736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:05.230702    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:05.394457 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:39:05.394407    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:39:05.468021 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:05.467988    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:08.993966 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:08.993929    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:14.910351 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:14.910307    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"]
Oct 31 19:39:14.917082 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:14.917051    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:14.968874 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:14.968836    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"]
Oct 31 19:39:14.986343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:14.986306    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5a4438\"/\"openshift-service-ca.crt\""
Oct 31 19:39:15.001774 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.001737    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-dshm\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.001942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.001859    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-model-cache\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.001942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.001900    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-home\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.002018 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.001954    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tls-certs\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.002018 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.001975    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldfrr\" (UniqueName: \"kubernetes.io/projected/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-kube-api-access-ldfrr\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.002119 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.002098    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tmp-dir\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.016857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.016827    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5a4438\"/\"kube-root-ca.crt\""
Oct 31 19:39:15.016967 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.016882    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a4438\"/\"default-dockercfg-6lxcz\""
Oct 31 19:39:15.019950 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.019931    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a4438\"/\"simulator-w6-5a4438-kserve-self-signed-certs\""
Oct 31 19:39:15.103622 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.103582    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tmp-dir\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.103622 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.103619    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-dshm\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.103896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.103693    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-model-cache\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.103896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.103714    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-home\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.103896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.103734    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tls-certs\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.103896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.103752    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldfrr\" (UniqueName: \"kubernetes.io/projected/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-kube-api-access-ldfrr\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.104128 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.104100    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tmp-dir\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.104190 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.104109    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-model-cache\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.104236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.104185    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-home\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.106114 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.106087    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-dshm\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.106363 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.106345    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tls-certs\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.161300 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.161219    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldfrr\" (UniqueName: \"kubernetes.io/projected/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-kube-api-access-ldfrr\") pod \"simulator-w6-5a4438-kserve-79b857bb6-9kbqt\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") " pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.230325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.230281    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:15.616457 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:39:15.616404    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb9ff14e_bcc8_44a2_9a4b_8691f58ce806.slice/crio-0befff9483fc7e5e85fabbc5f93447190b4e62e2f1f0102c0c9061a8afffea1c WatchSource:0}: Error finding container 0befff9483fc7e5e85fabbc5f93447190b4e62e2f1f0102c0c9061a8afffea1c: Status 404 returned error can't find the container with id 0befff9483fc7e5e85fabbc5f93447190b4e62e2f1f0102c0c9061a8afffea1c
Oct 31 19:39:15.618761 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:15.618736    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"]
Oct 31 19:39:16.346828 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:16.346782    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" event={"ID":"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806","Type":"ContainerStarted","Data":"7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b"}
Oct 31 19:39:16.347426 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:16.346837    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" event={"ID":"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806","Type":"ContainerStarted","Data":"0befff9483fc7e5e85fabbc5f93447190b4e62e2f1f0102c0c9061a8afffea1c"}
Oct 31 19:39:16.425829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:16.425795    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:16.463667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:16.463598    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podStartSLOduration=2.463581374 podStartE2EDuration="2.463581374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:16.461721613 +0000 UTC m=+1374.991255659" watchObservedRunningTime="2026-10-31 19:39:16.463581374 +0000 UTC m=+1374.993115428"
Oct 31 19:39:16.651025 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:16.650921    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:17.230489 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:17.230456    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:19.185389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:19.185356    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:19.877667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:19.877629    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:20.354667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:20.354614    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:39:21.039584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:21.039550    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:21.669078 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:21.669039    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:21.896100 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:21.896064    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:22.741758 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.741721    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"]
Oct 31 19:39:22.745914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.745893    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.818542 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.818512    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-093b97\"/\"openshift-service-ca.crt\""
Oct 31 19:39:22.864482 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.864446    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-093b97\"/\"kube-root-ca.crt\""
Oct 31 19:39:22.869037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.869018    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-093b97\"/\"default-dockercfg-kds7w\""
Oct 31 19:39:22.869037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.869028    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-093b97\"/\"simulator-w0-093b97-kserve-self-signed-certs\""
Oct 31 19:39:22.871171 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.871149    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"]
Oct 31 19:39:22.885538 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.885509    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-tmp-dir\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.885727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.885544    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-home\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.885727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.885710    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-model-cache\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.885847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.885756    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f16468-0585-46c2-a02d-09a958fc961f-tls-certs\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.885889 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.885854    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-dshm\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.885946 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.885927    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxz2w\" (UniqueName: \"kubernetes.io/projected/71f16468-0585-46c2-a02d-09a958fc961f-kube-api-access-dxz2w\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.986524 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.986487    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-model-cache\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.986736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.986532    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f16468-0585-46c2-a02d-09a958fc961f-tls-certs\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.986736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.986573    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-dshm\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.986736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.986665    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxz2w\" (UniqueName: \"kubernetes.io/projected/71f16468-0585-46c2-a02d-09a958fc961f-kube-api-access-dxz2w\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.986736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.986695    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-tmp-dir\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.986736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.986719    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-home\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.987016 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.986924    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-model-cache\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.987183 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.987157    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-tmp-dir\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.987259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.987167    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-home\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.989178 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.989153    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f16468-0585-46c2-a02d-09a958fc961f-tls-certs\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.989277 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.989205    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-dshm\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:22.993609 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:22.993529    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:23.050036 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:23.050004    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxz2w\" (UniqueName: \"kubernetes.io/projected/71f16468-0585-46c2-a02d-09a958fc961f-kube-api-access-dxz2w\") pod \"simulator-w0-093b97-kserve-8446dcf765-th55v\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") " pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:23.058790 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:23.058747    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:23.434563 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:39:23.434532    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71f16468_0585_46c2_a02d_09a958fc961f.slice/crio-4a3f22e70b879fc03a07df30fa8c5289c2ace77e23f8c718919f03a6ffb9ca6e WatchSource:0}: Error finding container 4a3f22e70b879fc03a07df30fa8c5289c2ace77e23f8c718919f03a6ffb9ca6e: Status 404 returned error can't find the container with id 4a3f22e70b879fc03a07df30fa8c5289c2ace77e23f8c718919f03a6ffb9ca6e
Oct 31 19:39:23.436432 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:23.436327    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"]
Oct 31 19:39:24.392319 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:24.392284    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" event={"ID":"71f16468-0585-46c2-a02d-09a958fc961f","Type":"ContainerStarted","Data":"55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d"}
Oct 31 19:39:24.392768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:24.392328    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" event={"ID":"71f16468-0585-46c2-a02d-09a958fc961f","Type":"ContainerStarted","Data":"4a3f22e70b879fc03a07df30fa8c5289c2ace77e23f8c718919f03a6ffb9ca6e"}
Oct 31 19:39:24.511330 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:24.511271    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podStartSLOduration=2.511257156 podStartE2EDuration="2.511257156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:24.50927922 +0000 UTC m=+1383.038813274" watchObservedRunningTime="2026-10-31 19:39:24.511257156 +0000 UTC m=+1383.040791208"
Oct 31 19:39:25.059467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:25.059426    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:25.231533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:25.231494    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:25.244841 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:25.244805    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:25.384117 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:25.384085    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:25.408614 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:25.408574    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:39:30.276120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:30.276081    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:31.204383 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.204283    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:31.432270 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.432230    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:31.665048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.665011    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:31.867357 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.867323    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"]
Oct 31 19:39:31.871340 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.871317    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.874359 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.874328    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-home\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.874491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.874363    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-model-cache\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.874491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.874387    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-dshm\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.874612 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.874496    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tmp-dir\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.874612 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.874542    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqd8\" (UniqueName: \"kubernetes.io/projected/847f87ee-2302-423f-a9f8-5a307a2e7b1a-kube-api-access-zvqd8\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.874738 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.874619    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tls-certs\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.926303 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.926220    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"]
Oct 31 19:39:31.973264 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.973233    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b5155c\"/\"default-dockercfg-9626k\""
Oct 31 19:39:31.973479 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.973346    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b5155c\"/\"simulator-w1-b5155c-kserve-self-signed-certs\""
Oct 31 19:39:31.974280 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.974254    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b5155c\"/\"openshift-service-ca.crt\""
Oct 31 19:39:31.974341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.974265    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b5155c\"/\"kube-root-ca.crt\""
Oct 31 19:39:31.975716 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.975698    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tls-certs\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.975776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.975758    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-home\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.975829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.975778    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-model-cache\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.975829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.975808    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-dshm\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.975930 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.975832    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tmp-dir\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.975930 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.975877    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqd8\" (UniqueName: \"kubernetes.io/projected/847f87ee-2302-423f-a9f8-5a307a2e7b1a-kube-api-access-zvqd8\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.976187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.976164    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-home\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.976288 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.976216    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tmp-dir\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.976348 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.976282    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-model-cache\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.978127 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.978108    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-dshm\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:31.978325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:31.978302    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tls-certs\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:32.050964 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:32.050922    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:32.087525 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:32.087491    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqd8\" (UniqueName: \"kubernetes.io/projected/847f87ee-2302-423f-a9f8-5a307a2e7b1a-kube-api-access-zvqd8\") pod \"simulator-w1-b5155c-kserve-5cbc7b8997-94jsv\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") " pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:32.183835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:32.183726    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:32.324595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:32.324546    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:32.559104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:32.559008    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:32.579278 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:39:32.579247    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod847f87ee_2302_423f_a9f8_5a307a2e7b1a.slice/crio-fcc4d95cd1139b72d32f011cb628e350cf8d2b55b256e6df827fef869d0a6ecb WatchSource:0}: Error finding container fcc4d95cd1139b72d32f011cb628e350cf8d2b55b256e6df827fef869d0a6ecb: Status 404 returned error can't find the container with id fcc4d95cd1139b72d32f011cb628e350cf8d2b55b256e6df827fef869d0a6ecb
Oct 31 19:39:32.582117 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:32.582090    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"]
Oct 31 19:39:33.059153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:33.059109    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:33.072127 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:33.072099    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:33.444169 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:33.444130    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" event={"ID":"847f87ee-2302-423f-a9f8-5a307a2e7b1a","Type":"ContainerStarted","Data":"f687613fdac29da3434086470ac1e80ca0e798df08f6b2b7e43bc7124d01843a"}
Oct 31 19:39:33.444169 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:33.444177    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" event={"ID":"847f87ee-2302-423f-a9f8-5a307a2e7b1a","Type":"ContainerStarted","Data":"fcc4d95cd1139b72d32f011cb628e350cf8d2b55b256e6df827fef869d0a6ecb"}
Oct 31 19:39:33.455476 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:33.455443    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:39:33.562540 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:33.562480    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podStartSLOduration=2.562460218 podStartE2EDuration="2.562460218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:33.560524387 +0000 UTC m=+1392.090058468" watchObservedRunningTime="2026-10-31 19:39:33.562460218 +0000 UTC m=+1392.091994270"
Oct 31 19:39:34.184460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:34.184422    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:35.393802 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:39:35.393764    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:39:35.559022 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:35.558986    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:37.493410 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:37.493366    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:39:40.399833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:40.399794    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:42.184714 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:42.184645    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:42.197459 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:42.197430    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:42.516881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:42.516797    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:39:43.771762 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.771726    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"]
Oct 31 19:39:43.774996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.774975    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:43.878457 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.878422    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"openshift-service-ca.crt\""
Oct 31 19:39:43.878457 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.878425    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"simulator-w5-570915-kserve-self-signed-certs\""
Oct 31 19:39:43.878775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.878492    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"kube-root-ca.crt\""
Oct 31 19:39:43.878775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.878683    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"default-dockercfg-96g4b\""
Oct 31 19:39:43.880525 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.880496    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"]
Oct 31 19:39:43.903869 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.903826    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-home\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:43.904043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.903881    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6494229e-f3e7-4798-b867-33dda48a521d-tls-certs\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:43.904043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.903992    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-model-cache\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:43.904150 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.904049    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4phg\" (UniqueName: \"kubernetes.io/projected/6494229e-f3e7-4798-b867-33dda48a521d-kube-api-access-r4phg\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:43.904209 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.904184    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-dshm\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:43.904272 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.904255    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-tmp-dir\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:43.957633 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:43.957594    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:44.004952 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.004917    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-home\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.005150 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.004973    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6494229e-f3e7-4798-b867-33dda48a521d-tls-certs\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.005150 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.005029    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-model-cache\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.005150 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.005091    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4phg\" (UniqueName: \"kubernetes.io/projected/6494229e-f3e7-4798-b867-33dda48a521d-kube-api-access-r4phg\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.005334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.005180    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-dshm\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.005334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.005264    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-tmp-dir\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.005464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.005395    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-home\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.005632 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.005607    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-model-cache\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.005772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.005703    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-tmp-dir\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.007429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.007399    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-dshm\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.007644 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.007625    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6494229e-f3e7-4798-b867-33dda48a521d-tls-certs\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.062351 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.062267    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4phg\" (UniqueName: \"kubernetes.io/projected/6494229e-f3e7-4798-b867-33dda48a521d-kube-api-access-r4phg\") pod \"simulator-w5-570915-kserve-d4854df47-vt2rb\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") " pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.088120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.088079    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:44.464503 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:39:44.464464    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6494229e_f3e7_4798_b867_33dda48a521d.slice/crio-05089b386f47e9e9fdb12790966f78f89878f79e7f71e1e79a715ec83809d509 WatchSource:0}: Error finding container 05089b386f47e9e9fdb12790966f78f89878f79e7f71e1e79a715ec83809d509: Status 404 returned error can't find the container with id 05089b386f47e9e9fdb12790966f78f89878f79e7f71e1e79a715ec83809d509
Oct 31 19:39:44.466439 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.466328    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"]
Oct 31 19:39:44.518332 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:44.518298    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" event={"ID":"6494229e-f3e7-4798-b867-33dda48a521d","Type":"ContainerStarted","Data":"05089b386f47e9e9fdb12790966f78f89878f79e7f71e1e79a715ec83809d509"}
Oct 31 19:39:45.272760 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:45.272723    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:45.525184 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:45.525091    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" event={"ID":"6494229e-f3e7-4798-b867-33dda48a521d","Type":"ContainerStarted","Data":"b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c"}
Oct 31 19:39:45.640455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:45.640401    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podStartSLOduration=2.64038525 podStartE2EDuration="2.64038525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:45.639746346 +0000 UTC m=+1404.169280420" watchObservedRunningTime="2026-10-31 19:39:45.64038525 +0000 UTC m=+1404.169919303"
Oct 31 19:39:46.088700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:46.088659    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:50.185829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:50.185793    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:51.798867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:51.798832    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:39:51.894734 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:51.894699    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:54.089265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:54.089227    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:54.103482 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:54.103440    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:54.587031 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:54.587002    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:39:55.068889 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:55.068784    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:59.886565 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:39:59.886518    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:00.137026 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.136941    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"]
Oct 31 19:40:00.141142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.141117    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.244315 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.244276    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7cfafd\"/\"default-dockercfg-pgfhv\""
Oct 31 19:40:00.244507 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.244404    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7cfafd\"/\"openshift-service-ca.crt\""
Oct 31 19:40:00.244507 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.244438    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7cfafd\"/\"kube-root-ca.crt\""
Oct 31 19:40:00.244507 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.244470    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7cfafd\"/\"simulator-w4-7cfafd-kserve-self-signed-certs\""
Oct 31 19:40:00.247983 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.247953    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"]
Oct 31 19:40:00.275179 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.275141    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-model-cache\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.275179 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.275183    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbsm8\" (UniqueName: \"kubernetes.io/projected/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-kube-api-access-jbsm8\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.275435 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.275258    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tls-certs\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.275435 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.275397    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tmp-dir\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.275522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.275507    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-dshm\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.275628 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.275612    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-home\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.324540 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.324495    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:00.376910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.376861    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-dshm\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.377120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.376956    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-home\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.377120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.377017    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-model-cache\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.377120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.377046    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbsm8\" (UniqueName: \"kubernetes.io/projected/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-kube-api-access-jbsm8\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.377120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.377082    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tls-certs\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.377339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.377150    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tmp-dir\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.377487 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.377460    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-home\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.377563 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.377539    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-model-cache\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.378256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.378225    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tmp-dir\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.379434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.379411    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-dshm\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.379975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.379954    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tls-certs\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.435454 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.435368    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbsm8\" (UniqueName: \"kubernetes.io/projected/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-kube-api-access-jbsm8\") pod \"simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") " pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.456278 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.456242    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:00.854911 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:40:00.854861    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e0cc5cc_1d03_4aab_970d_54b15f432bd4.slice/crio-20994f2f6a1a5c342f568766247e8b354e96c75537f6dcc971ecb014fcf6f4d5 WatchSource:0}: Error finding container 20994f2f6a1a5c342f568766247e8b354e96c75537f6dcc971ecb014fcf6f4d5: Status 404 returned error can't find the container with id 20994f2f6a1a5c342f568766247e8b354e96c75537f6dcc971ecb014fcf6f4d5
Oct 31 19:40:00.857217 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:00.857182    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"]
Oct 31 19:40:01.635301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:01.635264    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" event={"ID":"2e0cc5cc-1d03-4aab-970d-54b15f432bd4","Type":"ContainerStarted","Data":"3994e23e1a948b527e2966719a25dab0a5da12a34cd30b819006fbaf68dbd094"}
Oct 31 19:40:01.635301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:01.635308    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" event={"ID":"2e0cc5cc-1d03-4aab-970d-54b15f432bd4","Type":"ContainerStarted","Data":"20994f2f6a1a5c342f568766247e8b354e96c75537f6dcc971ecb014fcf6f4d5"}
Oct 31 19:40:01.750681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:01.750606    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podStartSLOduration=1.750591102 podStartE2EDuration="1.750591102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:01.750111064 +0000 UTC m=+1420.279645118" watchObservedRunningTime="2026-10-31 19:40:01.750591102 +0000 UTC m=+1420.280125155"
Oct 31 19:40:02.457170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:02.457135    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:04.788784 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:04.788748    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:05.394803 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:40:05.394755    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:40:07.281507 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:07.281466    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:40:08.891828 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:08.891795    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:09.764915 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:09.764881    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:10.457350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:10.457313    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:10.470340 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:10.470310    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:10.694325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:10.694298    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:40:14.681407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:14.681370    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:19.512481 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:19.512439    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:21.554294 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:21.554251    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:40:22.587866 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:22.587832    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:24.475753 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:24.475711    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:25.461879 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.461844    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"]
Oct 31 19:40:25.465937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.465914    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.520063 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.520026    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"]
Oct 31 19:40:25.534868 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.534827    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsrbl\" (UniqueName: \"kubernetes.io/projected/240043a0-2aea-451c-8658-d350ba4813e8-kube-api-access-lsrbl\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.535061 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.534901    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-home\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.535061 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.534953    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-dshm\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.535061 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.535040    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-tmp-dir\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.535234 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.535095    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-model-cache\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.535276 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.535240    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240043a0-2aea-451c-8658-d350ba4813e8-tls-certs\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.568102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.568059    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a4438\"/\"premium-w6-5a4438-kserve-self-signed-certs\""
Oct 31 19:40:25.632628 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.632592    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:25.635871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.635847    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240043a0-2aea-451c-8658-d350ba4813e8-tls-certs\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.635958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.635891    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsrbl\" (UniqueName: \"kubernetes.io/projected/240043a0-2aea-451c-8658-d350ba4813e8-kube-api-access-lsrbl\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.635958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.635935    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-home\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.635958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.635957    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-dshm\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.636112 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.635989    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-tmp-dir\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.636112 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.636022    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-model-cache\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.636481 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.636456    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-model-cache\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.636577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.636507    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-tmp-dir\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.636577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.636518    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-home\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.638281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.638262    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-dshm\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.638504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.638488    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240043a0-2aea-451c-8658-d350ba4813e8-tls-certs\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.693280 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.693241    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsrbl\" (UniqueName: \"kubernetes.io/projected/240043a0-2aea-451c-8658-d350ba4813e8-kube-api-access-lsrbl\") pod \"premium-w6-5a4438-kserve-7977db599-2zbxt\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") " pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:25.778447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:25.778354    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:26.168790 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:40:26.168749    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod240043a0_2aea_451c_8658_d350ba4813e8.slice/crio-c506b6f572f4c8ce752bd917efde286aa0c4749563d3f6838055d623fa5b538f WatchSource:0}: Error finding container c506b6f572f4c8ce752bd917efde286aa0c4749563d3f6838055d623fa5b538f: Status 404 returned error can't find the container with id c506b6f572f4c8ce752bd917efde286aa0c4749563d3f6838055d623fa5b538f
Oct 31 19:40:26.172239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:26.172210    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"]
Oct 31 19:40:26.778234 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:26.778192    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" event={"ID":"240043a0-2aea-451c-8658-d350ba4813e8","Type":"ContainerStarted","Data":"f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d"}
Oct 31 19:40:26.778636 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:26.778243    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" event={"ID":"240043a0-2aea-451c-8658-d350ba4813e8","Type":"ContainerStarted","Data":"c506b6f572f4c8ce752bd917efde286aa0c4749563d3f6838055d623fa5b538f"}
Oct 31 19:40:26.778636 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:26.778424    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:26.896747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:26.896695    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podStartSLOduration=1.896681961 podStartE2EDuration="1.896681961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:26.895684463 +0000 UTC m=+1445.425218513" watchObservedRunningTime="2026-10-31 19:40:26.896681961 +0000 UTC m=+1445.426216049"
Oct 31 19:40:29.369840 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:29.369791    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:34.278089 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:34.278049    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:35.394302 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:40:35.394264    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:40:35.779256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:35.779160    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:35.791890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:35.791859    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:35.848778 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:35.848750    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:40:39.198323 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:39.191513    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:40.449873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:40.449838    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:44.082224 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:44.082137    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:46.753854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:46.753811    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:40:49.000115 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:49.000074    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:53.883098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:53.883067    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:54.525009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.524973    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"]
Oct 31 19:40:54.528999 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.528978    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.561478 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.561436    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-model-cache\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.561693 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.561488    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tmp-dir\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.561693 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.561585    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-dshm\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.561693 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.561615    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tls-certs\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.561693 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.561641    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-home\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.561936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.561851    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plcp8\" (UniqueName: \"kubernetes.io/projected/b5c7c3d9-eda3-432e-91b6-40496d56f48e-kube-api-access-plcp8\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.575043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.574999    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"]
Oct 31 19:40:54.621712 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.621644    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b5155c\"/\"premium-w1-b5155c-kserve-self-signed-certs\""
Oct 31 19:40:54.663320 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.663282    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-dshm\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.663479 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.663326    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tls-certs\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.663479 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.663351    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-home\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.663727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.663699    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plcp8\" (UniqueName: \"kubernetes.io/projected/b5c7c3d9-eda3-432e-91b6-40496d56f48e-kube-api-access-plcp8\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.664331 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.663902    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-model-cache\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.664331 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.663944    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tmp-dir\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.664331 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.664282    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tmp-dir\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.664569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.664355    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-home\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.664569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.664389    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-model-cache\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.666460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.666433    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-dshm\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.667571 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.667550    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tls-certs\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.684508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.684476    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:54.723759 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.723730    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plcp8\" (UniqueName: \"kubernetes.io/projected/b5c7c3d9-eda3-432e-91b6-40496d56f48e-kube-api-access-plcp8\") pod \"premium-w1-b5155c-kserve-787d44c58d-4qg2t\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") " pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:54.842228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:54.842181    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:55.248329 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:40:55.248300    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5c7c3d9_eda3_432e_91b6_40496d56f48e.slice/crio-07f318708682646bca32ba864af218afb931ce55ddcd4234233445bb7a3fa88f WatchSource:0}: Error finding container 07f318708682646bca32ba864af218afb931ce55ddcd4234233445bb7a3fa88f: Status 404 returned error can't find the container with id 07f318708682646bca32ba864af218afb931ce55ddcd4234233445bb7a3fa88f
Oct 31 19:40:55.250285 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:55.250256    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"]
Oct 31 19:40:55.965012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:55.964972    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" event={"ID":"b5c7c3d9-eda3-432e-91b6-40496d56f48e","Type":"ContainerStarted","Data":"555092c5217a49251d0217cd1c7948409c78242ab128dc52e0d2bb001207f0cf"}
Oct 31 19:40:55.965012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:55.965018    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" event={"ID":"b5c7c3d9-eda3-432e-91b6-40496d56f48e","Type":"ContainerStarted","Data":"07f318708682646bca32ba864af218afb931ce55ddcd4234233445bb7a3fa88f"}
Oct 31 19:40:56.083047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:56.082991    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podStartSLOduration=2.082968145 podStartE2EDuration="2.082968145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:56.080525575 +0000 UTC m=+1474.610059628" watchObservedRunningTime="2026-10-31 19:40:56.082968145 +0000 UTC m=+1474.612502232"
Oct 31 19:40:56.843280 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:56.843245    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:40:58.772110 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:40:58.772068    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:03.579368 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:03.579278    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:04.842766 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:04.842721    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:41:04.855474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:04.855443    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:41:05.031634 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:05.031605    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:41:05.393335 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:41:05.393298    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:41:06.271160 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:06.271127    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:06.487694 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:06.487657    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:08.603773 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:08.603709    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:13.104619 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:13.104581    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:41:13.486597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:13.486507    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:18.384999 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:18.384957    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:19.850693 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:19.850663    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:21.861717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:21.861689    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:41:21.870238 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:21.870212    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:41:23.271037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:23.271000    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:28.077588 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:28.077547    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:29.904869 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:29.904832    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"]
Oct 31 19:41:29.909096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:29.909075    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:29.964336 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:29.964298    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"]
Oct 31 19:41:29.967320 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:29.967290    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-home\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:29.967470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:29.967336    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-tmp-dir\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:29.967546 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:29.967518    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-dshm\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:29.967680 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:29.967645    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt9sc\" (UniqueName: \"kubernetes.io/projected/da2d3282-0a38-4317-a213-9bba58d67932-kube-api-access-tt9sc\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:29.967754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:29.967733    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-model-cache\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:29.967868 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:29.967850    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da2d3282-0a38-4317-a213-9bba58d67932-tls-certs\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.010746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.010708    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7cfafd\"/\"premium-w4-7cfafd-kserve-self-signed-certs\""
Oct 31 19:41:30.070167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.070127    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-model-cache\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.070547 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.070488    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da2d3282-0a38-4317-a213-9bba58d67932-tls-certs\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.070931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.070908    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-home\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.071085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.071068    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-tmp-dir\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.071270 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.071243    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-dshm\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.071620 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.071567    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt9sc\" (UniqueName: \"kubernetes.io/projected/da2d3282-0a38-4317-a213-9bba58d67932-kube-api-access-tt9sc\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.071891 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.071737    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-tmp-dir\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.072102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.070588    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-model-cache\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.072102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.071266    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-home\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.073712 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.073688    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da2d3282-0a38-4317-a213-9bba58d67932-tls-certs\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.074745 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.074717    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-dshm\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.085390 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.085362    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:30.130234 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.130189    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt9sc\" (UniqueName: \"kubernetes.io/projected/da2d3282-0a38-4317-a213-9bba58d67932-kube-api-access-tt9sc\") pod \"premium-w4-7cfafd-kserve-85fb74694f-grr55\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") " pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.221211 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.221111    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:30.593495 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:41:30.593459    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda2d3282_0a38_4317_a213_9bba58d67932.slice/crio-223a1fb64193fd9cced2e55b333f9e1677ed94a61f930d1bb013168011b022c2 WatchSource:0}: Error finding container 223a1fb64193fd9cced2e55b333f9e1677ed94a61f930d1bb013168011b022c2: Status 404 returned error can't find the container with id 223a1fb64193fd9cced2e55b333f9e1677ed94a61f930d1bb013168011b022c2
Oct 31 19:41:30.595815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:30.595781    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"]
Oct 31 19:41:31.174526 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:31.174488    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" event={"ID":"da2d3282-0a38-4317-a213-9bba58d67932","Type":"ContainerStarted","Data":"23c102c5d5cdf1c335681ca3dfb545dccb9a6e15d2335e67f6ffdeb1800dc789"}
Oct 31 19:41:31.174974 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:31.174537    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" event={"ID":"da2d3282-0a38-4317-a213-9bba58d67932","Type":"ContainerStarted","Data":"223a1fb64193fd9cced2e55b333f9e1677ed94a61f930d1bb013168011b022c2"}
Oct 31 19:41:31.222142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:31.222106    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:31.290845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:31.290791    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podStartSLOduration=2.290776941 podStartE2EDuration="2.290776941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:31.288823306 +0000 UTC m=+1509.818357359" watchObservedRunningTime="2026-10-31 19:41:31.290776941 +0000 UTC m=+1509.820310994"
Oct 31 19:41:33.007383 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:33.007347    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:35.393118 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:41:35.393073    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:41:36.731060 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.731019    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"]
Oct 31 19:41:36.735627 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.735607    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:36.787867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.787832    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"]
Oct 31 19:41:36.788588 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.788571    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:41:36.835543 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.835512    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:41:36.835763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.835735    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:41:36.835855 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.835835    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:41:36.850906 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.850880    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e62d352-9kxnj\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:36.851060 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.850920    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e62d352-9kxnj\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:36.851060 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.851014    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87srw\" (UniqueName: \"kubernetes.io/projected/f9dfab44-81de-4f36-9826-03f638684e49-kube-api-access-87srw\") pod \"maas-api-revoke-sub-6e62d352-9kxnj\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:36.952256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.952216    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e62d352-9kxnj\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:36.952465 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.952269    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e62d352-9kxnj\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:36.952465 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.952309    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87srw\" (UniqueName: \"kubernetes.io/projected/f9dfab44-81de-4f36-9826-03f638684e49-kube-api-access-87srw\") pod \"maas-api-revoke-sub-6e62d352-9kxnj\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:36.953087 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.953063    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e62d352-9kxnj\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:36.954741 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:36.954720    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e62d352-9kxnj\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:37.009988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:37.009910    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87srw\" (UniqueName: \"kubernetes.io/projected/f9dfab44-81de-4f36-9826-03f638684e49-kube-api-access-87srw\") pod \"maas-api-revoke-sub-6e62d352-9kxnj\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:37.050439 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:37.050400    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:37.417623 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:37.417594    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"]
Oct 31 19:41:37.874822 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:37.874785    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:38.216789 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:38.216690    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj" event={"ID":"f9dfab44-81de-4f36-9826-03f638684e49","Type":"ContainerStarted","Data":"1eb6c00e0ffa8b45edebf96850ba75bc8b75f74ad6d59dccaaef2debe9601256"}
Oct 31 19:41:40.222040 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:40.221998    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:40.238871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:40.238830    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:40.255264 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:40.255228    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:41:40.419627 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:40.419503    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:41.328511 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:41.328475    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:41:42.800005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:42.799976    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:43.265843 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:43.265148    5006 generic.go:356] "Generic (PLEG): container finished" podID="f9dfab44-81de-4f36-9826-03f638684e49" containerID="ecd3d59ded8246a66db5dcb7a6e03866e458aefb4fab314821ca9e20d5ad8406" exitCode=0
Oct 31 19:41:43.265843 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:43.265256    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj" event={"ID":"f9dfab44-81de-4f36-9826-03f638684e49","Type":"ContainerDied","Data":"ecd3d59ded8246a66db5dcb7a6e03866e458aefb4fab314821ca9e20d5ad8406"}
Oct 31 19:41:44.426646 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:44.426618    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:44.550049 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:44.549963    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9dfab44-81de-4f36-9826-03f638684e49-kube-api-access-87srw\" (UniqueName: \"kubernetes.io/projected/f9dfab44-81de-4f36-9826-03f638684e49-kube-api-access-87srw\") pod \"f9dfab44-81de-4f36-9826-03f638684e49\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") "
Oct 31 19:41:44.550049 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:44.550038    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-xks-ca\") pod \"f9dfab44-81de-4f36-9826-03f638684e49\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") "
Oct 31 19:41:44.550293 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:44.550140    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-openshift-ca\") pod \"f9dfab44-81de-4f36-9826-03f638684e49\" (UID: \"f9dfab44-81de-4f36-9826-03f638684e49\") "
Oct 31 19:41:44.550739 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:44.550556    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-openshift-ca" pod "f9dfab44-81de-4f36-9826-03f638684e49" (UID: "f9dfab44-81de-4f36-9826-03f638684e49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:44.550855 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:44.550772    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.552898 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:44.552864    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-xks-ca" pod "f9dfab44-81de-4f36-9826-03f638684e49" (UID: "f9dfab44-81de-4f36-9826-03f638684e49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:41:44.552997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:44.552979    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9dfab44-81de-4f36-9826-03f638684e49-kube-api-access-87srw" pod "f9dfab44-81de-4f36-9826-03f638684e49" (UID: "f9dfab44-81de-4f36-9826-03f638684e49"). InnerVolumeSpecName "kube-api-access-87srw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:44.651990 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:44.651943    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87srw\" (UniqueName: \"kubernetes.io/projected/f9dfab44-81de-4f36-9826-03f638684e49-kube-api-access-87srw\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.651990 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:44.651989    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dfab44-81de-4f36-9826-03f638684e49-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:45.279245 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:45.279213    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"
Oct 31 19:41:45.279430 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:45.279244    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj" event={"ID":"f9dfab44-81de-4f36-9826-03f638684e49","Type":"ContainerDied","Data":"1eb6c00e0ffa8b45edebf96850ba75bc8b75f74ad6d59dccaaef2debe9601256"}
Oct 31 19:41:45.279430 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:45.279301    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eb6c00e0ffa8b45edebf96850ba75bc8b75f74ad6d59dccaaef2debe9601256"
Oct 31 19:41:46.184910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.184878    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"]
Oct 31 19:41:46.185363 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.185350    5006 cpu_manager.go:414] "containerMap: removing container" podUID="f9dfab44-81de-4f36-9826-03f638684e49" containerName="revoke-keys"
Oct 31 19:41:46.185407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.185364    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9dfab44-81de-4f36-9826-03f638684e49" containerName="revoke-keys"
Oct 31 19:41:46.185467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.185459    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9dfab44-81de-4f36-9826-03f638684e49" containerName="revoke-keys"
Oct 31 19:41:46.189036 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.189014    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:46.244439 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.244404    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"]
Oct 31 19:41:46.291140 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.291111    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:41:46.291140 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.291127    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:41:46.291140 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.291112    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:41:46.291998 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.291984    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:41:46.369880 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.369844    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-879cc0ff-b6s64\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:46.370071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.369892    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-879cc0ff-b6s64\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:46.370071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.370038    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlckf\" (UniqueName: \"kubernetes.io/projected/31f47dae-8f6d-481d-b818-6cd4d3980aee-kube-api-access-jlckf\") pod \"maas-api-revoke-sub-879cc0ff-b6s64\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:46.471021 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.470927    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-879cc0ff-b6s64\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:46.471021 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.470977    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-879cc0ff-b6s64\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:46.471246 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.471075    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlckf\" (UniqueName: \"kubernetes.io/projected/31f47dae-8f6d-481d-b818-6cd4d3980aee-kube-api-access-jlckf\") pod \"maas-api-revoke-sub-879cc0ff-b6s64\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:46.471603 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.471580    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-879cc0ff-b6s64\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:46.473312 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.473294    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-879cc0ff-b6s64\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:46.529147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.529115    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlckf\" (UniqueName: \"kubernetes.io/projected/31f47dae-8f6d-481d-b818-6cd4d3980aee-kube-api-access-jlckf\") pod \"maas-api-revoke-sub-879cc0ff-b6s64\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:46.802974 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:46.802883    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:47.169705 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:41:47.169671    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f47dae_8f6d_481d_b818_6cd4d3980aee.slice/crio-40e573c1cd5be2d8d982e719f723d050b66085588dd18b595924dd0be5a885e3 WatchSource:0}: Error finding container 40e573c1cd5be2d8d982e719f723d050b66085588dd18b595924dd0be5a885e3: Status 404 returned error can't find the container with id 40e573c1cd5be2d8d982e719f723d050b66085588dd18b595924dd0be5a885e3
Oct 31 19:41:47.174149 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:47.174123    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"]
Oct 31 19:41:47.291932 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:47.291901    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64" event={"ID":"31f47dae-8f6d-481d-b818-6cd4d3980aee","Type":"ContainerStarted","Data":"40e573c1cd5be2d8d982e719f723d050b66085588dd18b595924dd0be5a885e3"}
Oct 31 19:41:47.685919 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:47.685876    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:48.298943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:48.298911    5006 generic.go:356] "Generic (PLEG): container finished" podID="31f47dae-8f6d-481d-b818-6cd4d3980aee" containerID="6447e4ea3a665377584d15bc462d2eefb8a41b0b593c2f1615c199cb371036b8" exitCode=0
Oct 31 19:41:48.299405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:48.298991    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64" event={"ID":"31f47dae-8f6d-481d-b818-6cd4d3980aee","Type":"ContainerDied","Data":"6447e4ea3a665377584d15bc462d2eefb8a41b0b593c2f1615c199cb371036b8"}
Oct 31 19:41:49.454281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:49.454252    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:49.603412 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:49.603383    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-xks-ca\") pod \"31f47dae-8f6d-481d-b818-6cd4d3980aee\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") "
Oct 31 19:41:49.603569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:49.603555    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f47dae-8f6d-481d-b818-6cd4d3980aee-kube-api-access-jlckf\" (UniqueName: \"kubernetes.io/projected/31f47dae-8f6d-481d-b818-6cd4d3980aee-kube-api-access-jlckf\") pod \"31f47dae-8f6d-481d-b818-6cd4d3980aee\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") "
Oct 31 19:41:49.603611 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:49.603587    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-openshift-ca\") pod \"31f47dae-8f6d-481d-b818-6cd4d3980aee\" (UID: \"31f47dae-8f6d-481d-b818-6cd4d3980aee\") "
Oct 31 19:41:49.604000 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:49.603975    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-openshift-ca" pod "31f47dae-8f6d-481d-b818-6cd4d3980aee" (UID: "31f47dae-8f6d-481d-b818-6cd4d3980aee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:49.605603 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:49.605573    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-xks-ca" pod "31f47dae-8f6d-481d-b818-6cd4d3980aee" (UID: "31f47dae-8f6d-481d-b818-6cd4d3980aee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:41:49.605799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:49.605779    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f47dae-8f6d-481d-b818-6cd4d3980aee-kube-api-access-jlckf" pod "31f47dae-8f6d-481d-b818-6cd4d3980aee" (UID: "31f47dae-8f6d-481d-b818-6cd4d3980aee"). InnerVolumeSpecName "kube-api-access-jlckf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:49.705510 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:49.705477    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:49.705510 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:49.705503    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlckf\" (UniqueName: \"kubernetes.io/projected/31f47dae-8f6d-481d-b818-6cd4d3980aee-kube-api-access-jlckf\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:49.705510 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:49.705514    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f47dae-8f6d-481d-b818-6cd4d3980aee-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:50.311745 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:50.311704    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64" event={"ID":"31f47dae-8f6d-481d-b818-6cd4d3980aee","Type":"ContainerDied","Data":"40e573c1cd5be2d8d982e719f723d050b66085588dd18b595924dd0be5a885e3"}
Oct 31 19:41:50.311924 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:50.311767    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40e573c1cd5be2d8d982e719f723d050b66085588dd18b595924dd0be5a885e3"
Oct 31 19:41:50.311924 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:50.311722    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"
Oct 31 19:41:51.802020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:51.801987    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:52.102123 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.102085    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"]
Oct 31 19:41:52.102593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.102580    5006 cpu_manager.go:414] "containerMap: removing container" podUID="31f47dae-8f6d-481d-b818-6cd4d3980aee" containerName="revoke-keys"
Oct 31 19:41:52.102642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.102593    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f47dae-8f6d-481d-b818-6cd4d3980aee" containerName="revoke-keys"
Oct 31 19:41:52.102725 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.102714    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f47dae-8f6d-481d-b818-6cd4d3980aee" containerName="revoke-keys"
Oct 31 19:41:52.106245 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.106217    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.129013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.128979    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8f47ae-2x9nq\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.129212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.129188    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtm6r\" (UniqueName: \"kubernetes.io/projected/68908d9c-af34-4f37-9fd3-032f3e1e7535-kube-api-access-rtm6r\") pod \"maas-api-revoke-sub-5c8f47ae-2x9nq\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.129295 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.129243    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8f47ae-2x9nq\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.162122 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.162082    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"]
Oct 31 19:41:52.206503 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.206467    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:41:52.206685 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.206596    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:41:52.211235 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.211210    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:41:52.211373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.211318    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:41:52.230777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.230744    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtm6r\" (UniqueName: \"kubernetes.io/projected/68908d9c-af34-4f37-9fd3-032f3e1e7535-kube-api-access-rtm6r\") pod \"maas-api-revoke-sub-5c8f47ae-2x9nq\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.230881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.230783    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8f47ae-2x9nq\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.230881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.230837    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8f47ae-2x9nq\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.231542 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.231522    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8f47ae-2x9nq\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.233285 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.233266    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8f47ae-2x9nq\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.289195 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.289162    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtm6r\" (UniqueName: \"kubernetes.io/projected/68908d9c-af34-4f37-9fd3-032f3e1e7535-kube-api-access-rtm6r\") pod \"maas-api-revoke-sub-5c8f47ae-2x9nq\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.422900 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.422813    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:52.575939 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.575896    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:52.796309 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:41:52.796275    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68908d9c_af34_4f37_9fd3_032f3e1e7535.slice/crio-8f4fa9ea81ae06f6879a73d7407c3266fc2b97c983190419972f1b1a98f3ed23 WatchSource:0}: Error finding container 8f4fa9ea81ae06f6879a73d7407c3266fc2b97c983190419972f1b1a98f3ed23: Status 404 returned error can't find the container with id 8f4fa9ea81ae06f6879a73d7407c3266fc2b97c983190419972f1b1a98f3ed23
Oct 31 19:41:52.799535 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:52.799511    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"]
Oct 31 19:41:53.330166 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:53.330130    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq" event={"ID":"68908d9c-af34-4f37-9fd3-032f3e1e7535","Type":"ContainerStarted","Data":"140354c81a6764c63c08b68b482df36fe8ce4c887a1c3b09afa82f62032c772a"}
Oct 31 19:41:53.330580 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:53.330194    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq" event={"ID":"68908d9c-af34-4f37-9fd3-032f3e1e7535","Type":"ContainerStarted","Data":"8f4fa9ea81ae06f6879a73d7407c3266fc2b97c983190419972f1b1a98f3ed23"}
Oct 31 19:41:53.449797 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:53.449687    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq" podStartSLOduration=1.449667806 podStartE2EDuration="1.449667806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:53.448591937 +0000 UTC m=+1531.978125990" watchObservedRunningTime="2026-10-31 19:41:53.449667806 +0000 UTC m=+1531.979201860"
Oct 31 19:41:54.337033 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:54.336993    5006 generic.go:356] "Generic (PLEG): container finished" podID="68908d9c-af34-4f37-9fd3-032f3e1e7535" containerID="140354c81a6764c63c08b68b482df36fe8ce4c887a1c3b09afa82f62032c772a" exitCode=0
Oct 31 19:41:54.337429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:54.337079    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq" event={"ID":"68908d9c-af34-4f37-9fd3-032f3e1e7535","Type":"ContainerDied","Data":"140354c81a6764c63c08b68b482df36fe8ce4c887a1c3b09afa82f62032c772a"}
Oct 31 19:41:55.498260 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:55.498234    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:55.562794 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:55.562761    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68908d9c-af34-4f37-9fd3-032f3e1e7535-kube-api-access-rtm6r\" (UniqueName: \"kubernetes.io/projected/68908d9c-af34-4f37-9fd3-032f3e1e7535-kube-api-access-rtm6r\") pod \"68908d9c-af34-4f37-9fd3-032f3e1e7535\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") "
Oct 31 19:41:55.562794 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:55.562793    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-xks-ca\") pod \"68908d9c-af34-4f37-9fd3-032f3e1e7535\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") "
Oct 31 19:41:55.563053 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:55.562888    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-openshift-ca\") pod \"68908d9c-af34-4f37-9fd3-032f3e1e7535\" (UID: \"68908d9c-af34-4f37-9fd3-032f3e1e7535\") "
Oct 31 19:41:55.563310 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:55.563285    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-openshift-ca" pod "68908d9c-af34-4f37-9fd3-032f3e1e7535" (UID: "68908d9c-af34-4f37-9fd3-032f3e1e7535"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:55.565013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:55.564981    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68908d9c-af34-4f37-9fd3-032f3e1e7535-kube-api-access-rtm6r" pod "68908d9c-af34-4f37-9fd3-032f3e1e7535" (UID: "68908d9c-af34-4f37-9fd3-032f3e1e7535"). InnerVolumeSpecName "kube-api-access-rtm6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:55.565103 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:55.564990    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-xks-ca" pod "68908d9c-af34-4f37-9fd3-032f3e1e7535" (UID: "68908d9c-af34-4f37-9fd3-032f3e1e7535"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:41:55.664402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:55.664364    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtm6r\" (UniqueName: \"kubernetes.io/projected/68908d9c-af34-4f37-9fd3-032f3e1e7535-kube-api-access-rtm6r\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.664402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:55.664398    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.664402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:55.664410    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68908d9c-af34-4f37-9fd3-032f3e1e7535-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:56.350028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:56.349999    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"
Oct 31 19:41:56.350028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:56.350025    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq" event={"ID":"68908d9c-af34-4f37-9fd3-032f3e1e7535","Type":"ContainerDied","Data":"8f4fa9ea81ae06f6879a73d7407c3266fc2b97c983190419972f1b1a98f3ed23"}
Oct 31 19:41:56.350285 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:56.350073    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f4fa9ea81ae06f6879a73d7407c3266fc2b97c983190419972f1b1a98f3ed23"
Oct 31 19:41:57.592602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:41:57.592570    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:02.499873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:02.499823    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:05.396293 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:05.396257    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:05.468711 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:42:05.468673    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:42:05.613147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:05.613115    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:06.725423 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.725393    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"]
Oct 31 19:42:06.725977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.725961    5006 cpu_manager.go:414] "containerMap: removing container" podUID="68908d9c-af34-4f37-9fd3-032f3e1e7535" containerName="revoke-keys"
Oct 31 19:42:06.726029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.725978    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68908d9c-af34-4f37-9fd3-032f3e1e7535" containerName="revoke-keys"
Oct 31 19:42:06.726099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.726089    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="68908d9c-af34-4f37-9fd3-032f3e1e7535" containerName="revoke-keys"
Oct 31 19:42:06.729608 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.729592    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:06.783320 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.783291    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"]
Oct 31 19:42:06.802889 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.802856    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8a2f5d2-2nfvj\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:06.803061 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.802977    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgqmc\" (UniqueName: \"kubernetes.io/projected/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-kube-api-access-wgqmc\") pod \"maas-api-revoke-sub-b8a2f5d2-2nfvj\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:06.803135 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.803099    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8a2f5d2-2nfvj\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:06.831143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.831099    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:42:06.831143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.831127    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:42:06.831143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.831119    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:42:06.831429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.831174    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:42:06.904303 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.904270    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgqmc\" (UniqueName: \"kubernetes.io/projected/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-kube-api-access-wgqmc\") pod \"maas-api-revoke-sub-b8a2f5d2-2nfvj\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:06.904476 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.904333    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8a2f5d2-2nfvj\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:06.904476 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.904387    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8a2f5d2-2nfvj\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:06.904997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.904976    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8a2f5d2-2nfvj\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:06.906686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.906664    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8a2f5d2-2nfvj\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:06.971470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:06.971442    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgqmc\" (UniqueName: \"kubernetes.io/projected/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-kube-api-access-wgqmc\") pod \"maas-api-revoke-sub-b8a2f5d2-2nfvj\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:07.042535 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:07.042474    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:07.409708 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:07.409674    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7c9ad8_4428_40d2_a4bb_24dde49eab60.slice/crio-22743242c640514a01083c7ea32de2401ae69bc025b4298750992529190ce24e WatchSource:0}: Error finding container 22743242c640514a01083c7ea32de2401ae69bc025b4298750992529190ce24e: Status 404 returned error can't find the container with id 22743242c640514a01083c7ea32de2401ae69bc025b4298750992529190ce24e
Oct 31 19:42:07.412824 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:07.412793    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"]
Oct 31 19:42:07.422284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:07.422255    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj" event={"ID":"3b7c9ad8-4428-40d2-a4bb-24dde49eab60","Type":"ContainerStarted","Data":"22743242c640514a01083c7ea32de2401ae69bc025b4298750992529190ce24e"}
Oct 31 19:42:07.540251 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:07.540217    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:08.430419 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:08.430387    5006 generic.go:356] "Generic (PLEG): container finished" podID="3b7c9ad8-4428-40d2-a4bb-24dde49eab60" containerID="2d408947baf4f37504ef4cd20b2389c3ea5aaf965cdd2f540150f2f183bfebae" exitCode=0
Oct 31 19:42:08.430875 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:08.430467    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj" event={"ID":"3b7c9ad8-4428-40d2-a4bb-24dde49eab60","Type":"ContainerDied","Data":"2d408947baf4f37504ef4cd20b2389c3ea5aaf965cdd2f540150f2f183bfebae"}
Oct 31 19:42:09.604871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:09.604844    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:09.743305 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:09.743224    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-kube-api-access-wgqmc\" (UniqueName: \"kubernetes.io/projected/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-kube-api-access-wgqmc\") pod \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") "
Oct 31 19:42:09.743461 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:09.743324    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-xks-ca\") pod \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") "
Oct 31 19:42:09.743461 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:09.743388    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-openshift-ca\") pod \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\" (UID: \"3b7c9ad8-4428-40d2-a4bb-24dde49eab60\") "
Oct 31 19:42:09.743854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:09.743820    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-openshift-ca" pod "3b7c9ad8-4428-40d2-a4bb-24dde49eab60" (UID: "3b7c9ad8-4428-40d2-a4bb-24dde49eab60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:09.745517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:09.745485    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-xks-ca" pod "3b7c9ad8-4428-40d2-a4bb-24dde49eab60" (UID: "3b7c9ad8-4428-40d2-a4bb-24dde49eab60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:09.745942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:09.745919    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-kube-api-access-wgqmc" pod "3b7c9ad8-4428-40d2-a4bb-24dde49eab60" (UID: "3b7c9ad8-4428-40d2-a4bb-24dde49eab60"). InnerVolumeSpecName "kube-api-access-wgqmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:09.845116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:09.845081    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:09.845116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:09.845113    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:09.845116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:09.845123    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgqmc\" (UniqueName: \"kubernetes.io/projected/3b7c9ad8-4428-40d2-a4bb-24dde49eab60-kube-api-access-wgqmc\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:10.443806 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:10.443775    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"
Oct 31 19:42:10.443806 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:10.443783    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj" event={"ID":"3b7c9ad8-4428-40d2-a4bb-24dde49eab60","Type":"ContainerDied","Data":"22743242c640514a01083c7ea32de2401ae69bc025b4298750992529190ce24e"}
Oct 31 19:42:10.444023 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:10.443832    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22743242c640514a01083c7ea32de2401ae69bc025b4298750992529190ce24e"
Oct 31 19:42:12.489943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:12.489910    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:16.301025 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:16.300987    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:42:17.621243 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.621203    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"]
Oct 31 19:42:17.621927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.621906    5006 cpu_manager.go:414] "containerMap: removing container" podUID="3b7c9ad8-4428-40d2-a4bb-24dde49eab60" containerName="revoke-keys"
Oct 31 19:42:17.622028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.621928    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7c9ad8-4428-40d2-a4bb-24dde49eab60" containerName="revoke-keys"
Oct 31 19:42:17.622118 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.622105    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7c9ad8-4428-40d2-a4bb-24dde49eab60" containerName="revoke-keys"
Oct 31 19:42:17.630988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.630962    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:17.640477 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.640438    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd24f59-bmp9v\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:17.640618 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.640503    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd24f59-bmp9v\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:17.640697 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.640683    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pp7g\" (UniqueName: \"kubernetes.io/projected/eae2de8c-1742-488e-a1f7-140acbe32eab-kube-api-access-4pp7g\") pod \"maas-api-revoke-sub-3bd24f59-bmp9v\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:17.679403 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.679369    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"]
Oct 31 19:42:17.726101 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.726064    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:42:17.726281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.726132    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:42:17.726281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.726136    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:42:17.726281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.726133    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:42:17.735004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.734979    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:17.741399 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.741376    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pp7g\" (UniqueName: \"kubernetes.io/projected/eae2de8c-1742-488e-a1f7-140acbe32eab-kube-api-access-4pp7g\") pod \"maas-api-revoke-sub-3bd24f59-bmp9v\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:17.741486 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.741473    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd24f59-bmp9v\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:17.741527 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.741503    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd24f59-bmp9v\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:17.742132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.742112    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd24f59-bmp9v\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:17.743956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.743928    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd24f59-bmp9v\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:17.798724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.798626    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pp7g\" (UniqueName: \"kubernetes.io/projected/eae2de8c-1742-488e-a1f7-140acbe32eab-kube-api-access-4pp7g\") pod \"maas-api-revoke-sub-3bd24f59-bmp9v\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:17.944502 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:17.944410    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:18.312599 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:18.312550    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae2de8c_1742_488e_a1f7_140acbe32eab.slice/crio-fc4d6b5b5995f44dcc767be00b665ed429b4ca837f73cd7b4df8c389bccbb7ba WatchSource:0}: Error finding container fc4d6b5b5995f44dcc767be00b665ed429b4ca837f73cd7b4df8c389bccbb7ba: Status 404 returned error can't find the container with id fc4d6b5b5995f44dcc767be00b665ed429b4ca837f73cd7b4df8c389bccbb7ba
Oct 31 19:42:18.315360 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:18.315333    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"]
Oct 31 19:42:18.495717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:18.495680    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v" event={"ID":"eae2de8c-1742-488e-a1f7-140acbe32eab","Type":"ContainerStarted","Data":"fc4d6b5b5995f44dcc767be00b665ed429b4ca837f73cd7b4df8c389bccbb7ba"}
Oct 31 19:42:19.503440 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:19.503399    5006 generic.go:356] "Generic (PLEG): container finished" podID="eae2de8c-1742-488e-a1f7-140acbe32eab" containerID="940c5f0bc55803b055c46ceb50bc96466cc5c530760e73ad87476e6e94c27408" exitCode=0
Oct 31 19:42:19.503857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:19.503486    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v" event={"ID":"eae2de8c-1742-488e-a1f7-140acbe32eab","Type":"ContainerDied","Data":"940c5f0bc55803b055c46ceb50bc96466cc5c530760e73ad87476e6e94c27408"}
Oct 31 19:42:20.219545 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.219510    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:20.676673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.676627    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:20.775248 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.775155    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-xks-ca\") pod \"eae2de8c-1742-488e-a1f7-140acbe32eab\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") "
Oct 31 19:42:20.775248 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.775210    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae2de8c-1742-488e-a1f7-140acbe32eab-kube-api-access-4pp7g\" (UniqueName: \"kubernetes.io/projected/eae2de8c-1742-488e-a1f7-140acbe32eab-kube-api-access-4pp7g\") pod \"eae2de8c-1742-488e-a1f7-140acbe32eab\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") "
Oct 31 19:42:20.777337 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.777307    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-xks-ca" pod "eae2de8c-1742-488e-a1f7-140acbe32eab" (UID: "eae2de8c-1742-488e-a1f7-140acbe32eab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:20.777727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.777712    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae2de8c-1742-488e-a1f7-140acbe32eab-kube-api-access-4pp7g" pod "eae2de8c-1742-488e-a1f7-140acbe32eab" (UID: "eae2de8c-1742-488e-a1f7-140acbe32eab"). InnerVolumeSpecName "kube-api-access-4pp7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:20.875979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.875943    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-openshift-ca\") pod \"eae2de8c-1742-488e-a1f7-140acbe32eab\" (UID: \"eae2de8c-1742-488e-a1f7-140acbe32eab\") "
Oct 31 19:42:20.876247 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.876234    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:20.876292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.876251    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pp7g\" (UniqueName: \"kubernetes.io/projected/eae2de8c-1742-488e-a1f7-140acbe32eab-kube-api-access-4pp7g\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:20.876326 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.876293    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-openshift-ca" pod "eae2de8c-1742-488e-a1f7-140acbe32eab" (UID: "eae2de8c-1742-488e-a1f7-140acbe32eab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:20.976844 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:20.976802    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae2de8c-1742-488e-a1f7-140acbe32eab-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:21.323136 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:21.323095    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:21.532829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:21.532784    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v" event={"ID":"eae2de8c-1742-488e-a1f7-140acbe32eab","Type":"ContainerDied","Data":"fc4d6b5b5995f44dcc767be00b665ed429b4ca837f73cd7b4df8c389bccbb7ba"}
Oct 31 19:42:21.532997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:21.532884    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc4d6b5b5995f44dcc767be00b665ed429b4ca837f73cd7b4df8c389bccbb7ba"
Oct 31 19:42:21.532997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:21.532981    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"
Oct 31 19:42:22.614544 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:22.614509    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:25.007097 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.007061    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"]
Oct 31 19:42:25.007736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.007718    5006 cpu_manager.go:414] "containerMap: removing container" podUID="eae2de8c-1742-488e-a1f7-140acbe32eab" containerName="revoke-keys"
Oct 31 19:42:25.007788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.007737    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae2de8c-1742-488e-a1f7-140acbe32eab" containerName="revoke-keys"
Oct 31 19:42:25.007858 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.007848    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae2de8c-1742-488e-a1f7-140acbe32eab" containerName="revoke-keys"
Oct 31 19:42:25.012402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.012379    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.063422 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.063391    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"]
Oct 31 19:42:25.111214 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.111173    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:42:25.111214 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.111191    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:42:25.111471 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.111238    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:42:25.111471 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.111262    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:42:25.120970 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.120947    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeb43b24-xxx2d\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.121096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.121073    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtw7\" (UniqueName: \"kubernetes.io/projected/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-kube-api-access-vvtw7\") pod \"maas-api-revoke-sub-eeb43b24-xxx2d\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.121156 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.121142    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeb43b24-xxx2d\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.222298 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.222255    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeb43b24-xxx2d\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.222549 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.222387    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeb43b24-xxx2d\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.222549 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.222472    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtw7\" (UniqueName: \"kubernetes.io/projected/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-kube-api-access-vvtw7\") pod \"maas-api-revoke-sub-eeb43b24-xxx2d\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.223291 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.223265    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeb43b24-xxx2d\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.224920 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.224898    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeb43b24-xxx2d\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.280063 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.279991    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtw7\" (UniqueName: \"kubernetes.io/projected/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-kube-api-access-vvtw7\") pod \"maas-api-revoke-sub-eeb43b24-xxx2d\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.325844 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.325814    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:25.692490 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:25.692459    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0605535_ff87_4ce9_84b0_b40eb9cdce3f.slice/crio-f64821ba3671e0fa3666e906dde6775eaab986251d5735762097668e03c8e484 WatchSource:0}: Error finding container f64821ba3671e0fa3666e906dde6775eaab986251d5735762097668e03c8e484: Status 404 returned error can't find the container with id f64821ba3671e0fa3666e906dde6775eaab986251d5735762097668e03c8e484
Oct 31 19:42:25.695697 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:25.695674    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"]
Oct 31 19:42:26.563360 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:26.563324    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d" event={"ID":"e0605535-ff87-4ce9-84b0-b40eb9cdce3f","Type":"ContainerStarted","Data":"3d78396a9487d1fa96008ab05627e06bedf48e72b26d7f8abe28f8f107c027d2"}
Oct 31 19:42:26.563911 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:26.563417    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d" event={"ID":"e0605535-ff87-4ce9-84b0-b40eb9cdce3f","Type":"ContainerStarted","Data":"f64821ba3671e0fa3666e906dde6775eaab986251d5735762097668e03c8e484"}
Oct 31 19:42:26.680660 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:26.680596    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d" podStartSLOduration=2.68057778 podStartE2EDuration="2.68057778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:26.679568039 +0000 UTC m=+1565.209102093" watchObservedRunningTime="2026-10-31 19:42:26.68057778 +0000 UTC m=+1565.210111826"
Oct 31 19:42:27.563334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:27.563295    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:27.570411 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:27.570379    5006 generic.go:356] "Generic (PLEG): container finished" podID="e0605535-ff87-4ce9-84b0-b40eb9cdce3f" containerID="3d78396a9487d1fa96008ab05627e06bedf48e72b26d7f8abe28f8f107c027d2" exitCode=0
Oct 31 19:42:27.570901 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:27.570461    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d" event={"ID":"e0605535-ff87-4ce9-84b0-b40eb9cdce3f","Type":"ContainerDied","Data":"3d78396a9487d1fa96008ab05627e06bedf48e72b26d7f8abe28f8f107c027d2"}
Oct 31 19:42:28.731153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:28.731122    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:28.764561 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:28.764527    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-xks-ca\") pod \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") "
Oct 31 19:42:28.764811 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:28.764617    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-openshift-ca\") pod \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") "
Oct 31 19:42:28.764811 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:28.764670    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-kube-api-access-vvtw7\" (UniqueName: \"kubernetes.io/projected/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-kube-api-access-vvtw7\") pod \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\" (UID: \"e0605535-ff87-4ce9-84b0-b40eb9cdce3f\") "
Oct 31 19:42:28.764980 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:28.764952    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-openshift-ca" pod "e0605535-ff87-4ce9-84b0-b40eb9cdce3f" (UID: "e0605535-ff87-4ce9-84b0-b40eb9cdce3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:28.766706 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:28.766677    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-xks-ca" pod "e0605535-ff87-4ce9-84b0-b40eb9cdce3f" (UID: "e0605535-ff87-4ce9-84b0-b40eb9cdce3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:28.766807 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:28.766763    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-kube-api-access-vvtw7" pod "e0605535-ff87-4ce9-84b0-b40eb9cdce3f" (UID: "e0605535-ff87-4ce9-84b0-b40eb9cdce3f"). InnerVolumeSpecName "kube-api-access-vvtw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:28.866204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:28.866176    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:28.866204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:28.866200    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:28.866402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:28.866210    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtw7\" (UniqueName: \"kubernetes.io/projected/e0605535-ff87-4ce9-84b0-b40eb9cdce3f-kube-api-access-vvtw7\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:29.586090 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:29.586063    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"
Oct 31 19:42:29.586090 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:29.586082    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d" event={"ID":"e0605535-ff87-4ce9-84b0-b40eb9cdce3f","Type":"ContainerDied","Data":"f64821ba3671e0fa3666e906dde6775eaab986251d5735762097668e03c8e484"}
Oct 31 19:42:29.586299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:29.586123    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f64821ba3671e0fa3666e906dde6775eaab986251d5735762097668e03c8e484"
Oct 31 19:42:31.235217 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.235136    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"]
Oct 31 19:42:31.235623 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.235610    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e0605535-ff87-4ce9-84b0-b40eb9cdce3f" containerName="revoke-keys"
Oct 31 19:42:31.235690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.235624    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0605535-ff87-4ce9-84b0-b40eb9cdce3f" containerName="revoke-keys"
Oct 31 19:42:31.235783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.235773    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0605535-ff87-4ce9-84b0-b40eb9cdce3f" containerName="revoke-keys"
Oct 31 19:42:31.239259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.239241    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.292107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.292075    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d72ef74b-tjbrg\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.292302 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.292208    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d72ef74b-tjbrg\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.292302 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.292264    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpzf\" (UniqueName: \"kubernetes.io/projected/09967626-8fc7-4427-a695-8c1dbc2493ad-kube-api-access-7xpzf\") pod \"maas-api-revoke-sub-d72ef74b-tjbrg\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.293030 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.293005    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"]
Oct 31 19:42:31.339937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.339901    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:42:31.339937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.339922    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:42:31.339937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.339925    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:42:31.342415 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.342383    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:42:31.393359 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.393327    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d72ef74b-tjbrg\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.393547 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.393372    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpzf\" (UniqueName: \"kubernetes.io/projected/09967626-8fc7-4427-a695-8c1dbc2493ad-kube-api-access-7xpzf\") pod \"maas-api-revoke-sub-d72ef74b-tjbrg\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.393547 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.393469    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d72ef74b-tjbrg\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.394101 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.394073    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d72ef74b-tjbrg\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.395885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.395862    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d72ef74b-tjbrg\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.451353 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.451319    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpzf\" (UniqueName: \"kubernetes.io/projected/09967626-8fc7-4427-a695-8c1dbc2493ad-kube-api-access-7xpzf\") pod \"maas-api-revoke-sub-d72ef74b-tjbrg\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.556813 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.556728    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:31.926747 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:31.926711    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09967626_8fc7_4427_a695_8c1dbc2493ad.slice/crio-db24d90165ca6df2efe49453633ca07652cf874b8c9fe38441432609d3b9cb49 WatchSource:0}: Error finding container db24d90165ca6df2efe49453633ca07652cf874b8c9fe38441432609d3b9cb49: Status 404 returned error can't find the container with id db24d90165ca6df2efe49453633ca07652cf874b8c9fe38441432609d3b9cb49
Oct 31 19:42:31.929763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:31.929732    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"]
Oct 31 19:42:32.383766 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.383727    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:32.606291 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.606253    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg" event={"ID":"09967626-8fc7-4427-a695-8c1dbc2493ad","Type":"ContainerStarted","Data":"2dfa08e8f7364b8dbd09dc70662dd7aa2486133e7a9a73c982670ec8ad35fc21"}
Oct 31 19:42:32.606496 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.606312    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg" event={"ID":"09967626-8fc7-4427-a695-8c1dbc2493ad","Type":"ContainerStarted","Data":"db24d90165ca6df2efe49453633ca07652cf874b8c9fe38441432609d3b9cb49"}
Oct 31 19:42:32.611367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.611342    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"]
Oct 31 19:42:32.615343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.615323    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:32.670109 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.670076    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"]
Oct 31 19:42:32.707345 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.707307    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a9185c0-46cnw\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:32.707345 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.707347    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8tvv\" (UniqueName: \"kubernetes.io/projected/58c55589-c14a-4908-8bde-6cb47f40da3a-kube-api-access-j8tvv\") pod \"maas-api-revoke-sub-6a9185c0-46cnw\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:32.707611 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.707531    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a9185c0-46cnw\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:32.722288 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.722224    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg" podStartSLOduration=1.722205048 podStartE2EDuration="1.722205048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:32.721103342 +0000 UTC m=+1571.250637397" watchObservedRunningTime="2026-10-31 19:42:32.722205048 +0000 UTC m=+1571.251739102"
Oct 31 19:42:32.808921 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.808897    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a9185c0-46cnw\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:32.809026 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.809012    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a9185c0-46cnw\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:32.809074 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.809033    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8tvv\" (UniqueName: \"kubernetes.io/projected/58c55589-c14a-4908-8bde-6cb47f40da3a-kube-api-access-j8tvv\") pod \"maas-api-revoke-sub-6a9185c0-46cnw\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:32.809593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.809568    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a9185c0-46cnw\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:32.811272 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.811247    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a9185c0-46cnw\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:32.866718 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.866635    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8tvv\" (UniqueName: \"kubernetes.io/projected/58c55589-c14a-4908-8bde-6cb47f40da3a-kube-api-access-j8tvv\") pod \"maas-api-revoke-sub-6a9185c0-46cnw\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:32.929704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:32.929617    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:33.301382 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:33.301338    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c55589_c14a_4908_8bde_6cb47f40da3a.slice/crio-7494e5563463724720272fc29aacf084f4d01f1a88ac4b8b07bcbbddf1ef7054 WatchSource:0}: Error finding container 7494e5563463724720272fc29aacf084f4d01f1a88ac4b8b07bcbbddf1ef7054: Status 404 returned error can't find the container with id 7494e5563463724720272fc29aacf084f4d01f1a88ac4b8b07bcbbddf1ef7054
Oct 31 19:42:33.302968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.302889    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"]
Oct 31 19:42:33.613729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.613702    5006 generic.go:356] "Generic (PLEG): container finished" podID="09967626-8fc7-4427-a695-8c1dbc2493ad" containerID="2dfa08e8f7364b8dbd09dc70662dd7aa2486133e7a9a73c982670ec8ad35fc21" exitCode=0
Oct 31 19:42:33.614076 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.613787    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg" event={"ID":"09967626-8fc7-4427-a695-8c1dbc2493ad","Type":"ContainerDied","Data":"2dfa08e8f7364b8dbd09dc70662dd7aa2486133e7a9a73c982670ec8ad35fc21"}
Oct 31 19:42:33.615374 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.615352    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw" event={"ID":"58c55589-c14a-4908-8bde-6cb47f40da3a","Type":"ContainerStarted","Data":"7494e5563463724720272fc29aacf084f4d01f1a88ac4b8b07bcbbddf1ef7054"}
Oct 31 19:42:33.844023 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.843965    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw" podStartSLOduration=1.843950641 podStartE2EDuration="1.843950641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:33.840801194 +0000 UTC m=+1572.370335247" watchObservedRunningTime="2026-10-31 19:42:33.843950641 +0000 UTC m=+1572.373484693"
Oct 31 19:42:33.867740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.867620    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"]
Oct 31 19:42:33.871831 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.871805    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:33.921127 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.921087    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68f9t\" (UniqueName: \"kubernetes.io/projected/7796af89-4ef9-450a-8b69-2cc56820df8c-kube-api-access-68f9t\") pod \"maas-api-revoke-sub-fae8856a-gw7cm\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:33.921295 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.921159    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae8856a-gw7cm\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:33.921351 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.921314    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae8856a-gw7cm\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:33.925433 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:33.925406    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"]
Oct 31 19:42:34.022226 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.022184    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68f9t\" (UniqueName: \"kubernetes.io/projected/7796af89-4ef9-450a-8b69-2cc56820df8c-kube-api-access-68f9t\") pod \"maas-api-revoke-sub-fae8856a-gw7cm\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:34.022400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.022339    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae8856a-gw7cm\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:34.022490 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.022463    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae8856a-gw7cm\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:34.023162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.023133    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae8856a-gw7cm\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:34.024834 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.024812    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae8856a-gw7cm\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:34.080636 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.080601    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68f9t\" (UniqueName: \"kubernetes.io/projected/7796af89-4ef9-450a-8b69-2cc56820df8c-kube-api-access-68f9t\") pod \"maas-api-revoke-sub-fae8856a-gw7cm\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:34.185712 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.185615    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:34.557254 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:34.557220    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7796af89_4ef9_450a_8b69_2cc56820df8c.slice/crio-7ff9366d9ad323a74fda7c3dcaaa6056c4cc5dcf3f97390709fd1d421eae1a80 WatchSource:0}: Error finding container 7ff9366d9ad323a74fda7c3dcaaa6056c4cc5dcf3f97390709fd1d421eae1a80: Status 404 returned error can't find the container with id 7ff9366d9ad323a74fda7c3dcaaa6056c4cc5dcf3f97390709fd1d421eae1a80
Oct 31 19:42:34.559910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.559878    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"]
Oct 31 19:42:34.623168 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.623129    5006 generic.go:356] "Generic (PLEG): container finished" podID="58c55589-c14a-4908-8bde-6cb47f40da3a" containerID="0003ca0182ea9966f51975935f47bf9dfe6910e01b65d6190121fe51a81b0089" exitCode=0
Oct 31 19:42:34.623564 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.623217    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw" event={"ID":"58c55589-c14a-4908-8bde-6cb47f40da3a","Type":"ContainerDied","Data":"0003ca0182ea9966f51975935f47bf9dfe6910e01b65d6190121fe51a81b0089"}
Oct 31 19:42:34.624521 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.624486    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm" event={"ID":"7796af89-4ef9-450a-8b69-2cc56820df8c","Type":"ContainerStarted","Data":"7ff9366d9ad323a74fda7c3dcaaa6056c4cc5dcf3f97390709fd1d421eae1a80"}
Oct 31 19:42:34.786530 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.786496    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:34.829752 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.829720    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-openshift-ca\") pod \"09967626-8fc7-4427-a695-8c1dbc2493ad\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") "
Oct 31 19:42:34.829907 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.829894    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-xks-ca\") pod \"09967626-8fc7-4427-a695-8c1dbc2493ad\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") "
Oct 31 19:42:34.829948 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.829916    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09967626-8fc7-4427-a695-8c1dbc2493ad-kube-api-access-7xpzf\" (UniqueName: \"kubernetes.io/projected/09967626-8fc7-4427-a695-8c1dbc2493ad-kube-api-access-7xpzf\") pod \"09967626-8fc7-4427-a695-8c1dbc2493ad\" (UID: \"09967626-8fc7-4427-a695-8c1dbc2493ad\") "
Oct 31 19:42:34.830097 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.830074    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-openshift-ca" pod "09967626-8fc7-4427-a695-8c1dbc2493ad" (UID: "09967626-8fc7-4427-a695-8c1dbc2493ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:34.830272 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.830246    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:34.832198 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.832172    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-xks-ca" pod "09967626-8fc7-4427-a695-8c1dbc2493ad" (UID: "09967626-8fc7-4427-a695-8c1dbc2493ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:34.832297 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.832222    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09967626-8fc7-4427-a695-8c1dbc2493ad-kube-api-access-7xpzf" pod "09967626-8fc7-4427-a695-8c1dbc2493ad" (UID: "09967626-8fc7-4427-a695-8c1dbc2493ad"). InnerVolumeSpecName "kube-api-access-7xpzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:34.931318 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.931287    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09967626-8fc7-4427-a695-8c1dbc2493ad-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:34.931318 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:34.931317    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xpzf\" (UniqueName: \"kubernetes.io/projected/09967626-8fc7-4427-a695-8c1dbc2493ad-kube-api-access-7xpzf\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:35.393388 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:42:35.393349    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:42:35.631468 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.631436    5006 generic.go:356] "Generic (PLEG): container finished" podID="7796af89-4ef9-450a-8b69-2cc56820df8c" containerID="0de906dbe60c7f429196853e72a4de06f2924b9f32345ffd4e4318e9d5bcb682" exitCode=0
Oct 31 19:42:35.631896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.631522    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm" event={"ID":"7796af89-4ef9-450a-8b69-2cc56820df8c","Type":"ContainerDied","Data":"0de906dbe60c7f429196853e72a4de06f2924b9f32345ffd4e4318e9d5bcb682"}
Oct 31 19:42:35.632883 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.632866    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"
Oct 31 19:42:35.632981 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.632878    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg" event={"ID":"09967626-8fc7-4427-a695-8c1dbc2493ad","Type":"ContainerDied","Data":"db24d90165ca6df2efe49453633ca07652cf874b8c9fe38441432609d3b9cb49"}
Oct 31 19:42:35.632981 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.632918    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db24d90165ca6df2efe49453633ca07652cf874b8c9fe38441432609d3b9cb49"
Oct 31 19:42:35.788815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.788789    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:35.840113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.840079    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c55589-c14a-4908-8bde-6cb47f40da3a-kube-api-access-j8tvv\" (UniqueName: \"kubernetes.io/projected/58c55589-c14a-4908-8bde-6cb47f40da3a-kube-api-access-j8tvv\") pod \"58c55589-c14a-4908-8bde-6cb47f40da3a\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") "
Oct 31 19:42:35.840281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.840191    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-xks-ca\") pod \"58c55589-c14a-4908-8bde-6cb47f40da3a\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") "
Oct 31 19:42:35.840281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.840218    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-openshift-ca\") pod \"58c55589-c14a-4908-8bde-6cb47f40da3a\" (UID: \"58c55589-c14a-4908-8bde-6cb47f40da3a\") "
Oct 31 19:42:35.840645 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.840603    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-openshift-ca" pod "58c55589-c14a-4908-8bde-6cb47f40da3a" (UID: "58c55589-c14a-4908-8bde-6cb47f40da3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:35.842287 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.842265    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-xks-ca" pod "58c55589-c14a-4908-8bde-6cb47f40da3a" (UID: "58c55589-c14a-4908-8bde-6cb47f40da3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:35.842403 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.842393    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c55589-c14a-4908-8bde-6cb47f40da3a-kube-api-access-j8tvv" pod "58c55589-c14a-4908-8bde-6cb47f40da3a" (UID: "58c55589-c14a-4908-8bde-6cb47f40da3a"). InnerVolumeSpecName "kube-api-access-j8tvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:35.942227 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.942146    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8tvv\" (UniqueName: \"kubernetes.io/projected/58c55589-c14a-4908-8bde-6cb47f40da3a-kube-api-access-j8tvv\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:35.942227 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.942174    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:35.942227 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:35.942184    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c55589-c14a-4908-8bde-6cb47f40da3a-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:36.640300 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.640260    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw" event={"ID":"58c55589-c14a-4908-8bde-6cb47f40da3a","Type":"ContainerDied","Data":"7494e5563463724720272fc29aacf084f4d01f1a88ac4b8b07bcbbddf1ef7054"}
Oct 31 19:42:36.640775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.640332    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7494e5563463724720272fc29aacf084f4d01f1a88ac4b8b07bcbbddf1ef7054"
Oct 31 19:42:36.640775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.640273    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"
Oct 31 19:42:36.801900 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.801877    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:36.850338 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.850305    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7796af89-4ef9-450a-8b69-2cc56820df8c-kube-api-access-68f9t\" (UniqueName: \"kubernetes.io/projected/7796af89-4ef9-450a-8b69-2cc56820df8c-kube-api-access-68f9t\") pod \"7796af89-4ef9-450a-8b69-2cc56820df8c\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") "
Oct 31 19:42:36.850578 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.850389    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-xks-ca\") pod \"7796af89-4ef9-450a-8b69-2cc56820df8c\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") "
Oct 31 19:42:36.850578 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.850441    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-openshift-ca\") pod \"7796af89-4ef9-450a-8b69-2cc56820df8c\" (UID: \"7796af89-4ef9-450a-8b69-2cc56820df8c\") "
Oct 31 19:42:36.850880 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.850848    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-openshift-ca" pod "7796af89-4ef9-450a-8b69-2cc56820df8c" (UID: "7796af89-4ef9-450a-8b69-2cc56820df8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:36.852585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.852558    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7796af89-4ef9-450a-8b69-2cc56820df8c-kube-api-access-68f9t" pod "7796af89-4ef9-450a-8b69-2cc56820df8c" (UID: "7796af89-4ef9-450a-8b69-2cc56820df8c"). InnerVolumeSpecName "kube-api-access-68f9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:36.852585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.852561    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-xks-ca" pod "7796af89-4ef9-450a-8b69-2cc56820df8c" (UID: "7796af89-4ef9-450a-8b69-2cc56820df8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:36.951570 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.951485    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68f9t\" (UniqueName: \"kubernetes.io/projected/7796af89-4ef9-450a-8b69-2cc56820df8c-kube-api-access-68f9t\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:36.951570 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.951514    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:36.951570 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:36.951524    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7796af89-4ef9-450a-8b69-2cc56820df8c-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:37.181427 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:37.181396    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:37.648511 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:37.648481    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"
Oct 31 19:42:37.648511 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:37.648507    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm" event={"ID":"7796af89-4ef9-450a-8b69-2cc56820df8c","Type":"ContainerDied","Data":"7ff9366d9ad323a74fda7c3dcaaa6056c4cc5dcf3f97390709fd1d421eae1a80"}
Oct 31 19:42:37.649017 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:37.648555    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ff9366d9ad323a74fda7c3dcaaa6056c4cc5dcf3f97390709fd1d421eae1a80"
Oct 31 19:42:40.731850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:40.731790    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ks646 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:40.732482 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:40.732427    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:42.189142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:42.189106    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:44.386426 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.386389    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"]
Oct 31 19:42:44.386940 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.386923    5006 cpu_manager.go:414] "containerMap: removing container" podUID="09967626-8fc7-4427-a695-8c1dbc2493ad" containerName="revoke-keys"
Oct 31 19:42:44.386987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.386941    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09967626-8fc7-4427-a695-8c1dbc2493ad" containerName="revoke-keys"
Oct 31 19:42:44.386987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.386959    5006 cpu_manager.go:414] "containerMap: removing container" podUID="58c55589-c14a-4908-8bde-6cb47f40da3a" containerName="revoke-keys"
Oct 31 19:42:44.386987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.386964    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c55589-c14a-4908-8bde-6cb47f40da3a" containerName="revoke-keys"
Oct 31 19:42:44.386987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.386986    5006 cpu_manager.go:414] "containerMap: removing container" podUID="7796af89-4ef9-450a-8b69-2cc56820df8c" containerName="revoke-keys"
Oct 31 19:42:44.387108 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.386991    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7796af89-4ef9-450a-8b69-2cc56820df8c" containerName="revoke-keys"
Oct 31 19:42:44.387108 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.387101    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="7796af89-4ef9-450a-8b69-2cc56820df8c" containerName="revoke-keys"
Oct 31 19:42:44.387170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.387112    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="09967626-8fc7-4427-a695-8c1dbc2493ad" containerName="revoke-keys"
Oct 31 19:42:44.387170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.387119    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c55589-c14a-4908-8bde-6cb47f40da3a" containerName="revoke-keys"
Oct 31 19:42:44.390101 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.390085    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:44.443182 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.443148    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"]
Oct 31 19:42:44.449204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.449161    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdpn9\" (UniqueName: \"kubernetes.io/projected/09a8f083-07f6-4101-bbc9-76b9797e5593-kube-api-access-mdpn9\") pod \"maas-api-revoke-sub-316dc445-mwzdd\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:44.449418 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.449309    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-316dc445-mwzdd\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:44.449479 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.449427    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-316dc445-mwzdd\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:44.490445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.490412    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:42:44.490445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.490434    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:42:44.490707 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.490515    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:42:44.493003 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.492983    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:42:44.550512 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.550476    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-316dc445-mwzdd\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:44.550704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.550566    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdpn9\" (UniqueName: \"kubernetes.io/projected/09a8f083-07f6-4101-bbc9-76b9797e5593-kube-api-access-mdpn9\") pod \"maas-api-revoke-sub-316dc445-mwzdd\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:44.550704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.550613    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-316dc445-mwzdd\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:44.551190 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.551165    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-316dc445-mwzdd\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:44.553094 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.553063    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-316dc445-mwzdd\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:44.608132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.608104    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdpn9\" (UniqueName: \"kubernetes.io/projected/09a8f083-07f6-4101-bbc9-76b9797e5593-kube-api-access-mdpn9\") pod \"maas-api-revoke-sub-316dc445-mwzdd\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:44.704018 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:44.703927    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:45.082562 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:45.082522    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"]
Oct 31 19:42:45.084267 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:45.084238    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a8f083_07f6_4101_bbc9_76b9797e5593.slice/crio-2cf1c2e6e63c364bf0669e653093635d06e76e9d2ec577ea33553d92a651a99f WatchSource:0}: Error finding container 2cf1c2e6e63c364bf0669e653093635d06e76e9d2ec577ea33553d92a651a99f: Status 404 returned error can't find the container with id 2cf1c2e6e63c364bf0669e653093635d06e76e9d2ec577ea33553d92a651a99f
Oct 31 19:42:45.722179 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:45.722095    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd" event={"ID":"09a8f083-07f6-4101-bbc9-76b9797e5593","Type":"ContainerStarted","Data":"e8b63fce1f8488bd4c5ccf45cd4542eafb2ccb73a6cda23ecbf5fb3f22f8a7b7"}
Oct 31 19:42:45.722179 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:45.722155    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd" event={"ID":"09a8f083-07f6-4101-bbc9-76b9797e5593","Type":"ContainerStarted","Data":"2cf1c2e6e63c364bf0669e653093635d06e76e9d2ec577ea33553d92a651a99f"}
Oct 31 19:42:45.838599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:45.838543    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd" podStartSLOduration=1.8385278139999999 podStartE2EDuration="1.838527814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:45.837860848 +0000 UTC m=+1584.367394922" watchObservedRunningTime="2026-10-31 19:42:45.838527814 +0000 UTC m=+1584.368061867"
Oct 31 19:42:46.730040 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:46.730006    5006 generic.go:356] "Generic (PLEG): container finished" podID="09a8f083-07f6-4101-bbc9-76b9797e5593" containerID="e8b63fce1f8488bd4c5ccf45cd4542eafb2ccb73a6cda23ecbf5fb3f22f8a7b7" exitCode=0
Oct 31 19:42:46.730435 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:46.730086    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd" event={"ID":"09a8f083-07f6-4101-bbc9-76b9797e5593","Type":"ContainerDied","Data":"e8b63fce1f8488bd4c5ccf45cd4542eafb2ccb73a6cda23ecbf5fb3f22f8a7b7"}
Oct 31 19:42:47.087087 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:47.087032    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:47.914067 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:47.914035    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:47.996455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:47.996359    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a8f083-07f6-4101-bbc9-76b9797e5593-kube-api-access-mdpn9\" (UniqueName: \"kubernetes.io/projected/09a8f083-07f6-4101-bbc9-76b9797e5593-kube-api-access-mdpn9\") pod \"09a8f083-07f6-4101-bbc9-76b9797e5593\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") "
Oct 31 19:42:47.996455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:47.996429    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-xks-ca\") pod \"09a8f083-07f6-4101-bbc9-76b9797e5593\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") "
Oct 31 19:42:47.996705 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:47.996520    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-openshift-ca\") pod \"09a8f083-07f6-4101-bbc9-76b9797e5593\" (UID: \"09a8f083-07f6-4101-bbc9-76b9797e5593\") "
Oct 31 19:42:47.996887 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:47.996857    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-openshift-ca" pod "09a8f083-07f6-4101-bbc9-76b9797e5593" (UID: "09a8f083-07f6-4101-bbc9-76b9797e5593"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:47.998629 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:47.998599    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-xks-ca" pod "09a8f083-07f6-4101-bbc9-76b9797e5593" (UID: "09a8f083-07f6-4101-bbc9-76b9797e5593"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:47.998754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:47.998730    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a8f083-07f6-4101-bbc9-76b9797e5593-kube-api-access-mdpn9" pod "09a8f083-07f6-4101-bbc9-76b9797e5593" (UID: "09a8f083-07f6-4101-bbc9-76b9797e5593"). InnerVolumeSpecName "kube-api-access-mdpn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:48.097621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:48.097585    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdpn9\" (UniqueName: \"kubernetes.io/projected/09a8f083-07f6-4101-bbc9-76b9797e5593-kube-api-access-mdpn9\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:48.097621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:48.097616    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:48.097621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:48.097627    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a8f083-07f6-4101-bbc9-76b9797e5593-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:48.744412 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:48.744374    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"
Oct 31 19:42:48.744412 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:48.744396    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd" event={"ID":"09a8f083-07f6-4101-bbc9-76b9797e5593","Type":"ContainerDied","Data":"2cf1c2e6e63c364bf0669e653093635d06e76e9d2ec577ea33553d92a651a99f"}
Oct 31 19:42:48.744619 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:48.744442    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf1c2e6e63c364bf0669e653093635d06e76e9d2ec577ea33553d92a651a99f"
Oct 31 19:42:49.856675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.856502    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"]
Oct 31 19:42:49.857746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.857342    5006 cpu_manager.go:414] "containerMap: removing container" podUID="09a8f083-07f6-4101-bbc9-76b9797e5593" containerName="revoke-keys"
Oct 31 19:42:49.857746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.857368    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a8f083-07f6-4101-bbc9-76b9797e5593" containerName="revoke-keys"
Oct 31 19:42:49.857746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.857492    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a8f083-07f6-4101-bbc9-76b9797e5593" containerName="revoke-keys"
Oct 31 19:42:49.862887 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.861672    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:49.911056 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.911019    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"]
Oct 31 19:42:49.917484 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.917448    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k857k\" (UniqueName: \"kubernetes.io/projected/1406693d-432d-4f84-9d81-d0bdc062f65c-kube-api-access-k857k\") pod \"maas-api-revoke-sub-955eb881-bt54f\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:49.917685 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.917499    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955eb881-bt54f\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:49.917761 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.917690    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955eb881-bt54f\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:49.958407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.958374    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:42:49.958407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.958385    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:42:49.958407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.958385    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:42:49.964809 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:49.964792    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:42:50.018450 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.018412    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k857k\" (UniqueName: \"kubernetes.io/projected/1406693d-432d-4f84-9d81-d0bdc062f65c-kube-api-access-k857k\") pod \"maas-api-revoke-sub-955eb881-bt54f\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:50.018450 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.018454    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955eb881-bt54f\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:50.018752 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.018529    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955eb881-bt54f\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:50.019239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.019206    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955eb881-bt54f\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:50.021044 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.021022    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955eb881-bt54f\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:50.076969 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.076936    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k857k\" (UniqueName: \"kubernetes.io/projected/1406693d-432d-4f84-9d81-d0bdc062f65c-kube-api-access-k857k\") pod \"maas-api-revoke-sub-955eb881-bt54f\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:50.183572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.183487    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:50.230310 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.230271    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"]
Oct 31 19:42:50.235294 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.235261    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.286601 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.286553    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"]
Oct 31 19:42:50.321777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.321746    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4001af4c-pmtrz\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.321977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.321820    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvw8g\" (UniqueName: \"kubernetes.io/projected/cda96c2d-0405-4840-9836-6357f68f9b68-kube-api-access-xvw8g\") pod \"maas-api-revoke-sub-4001af4c-pmtrz\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.321977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.321956    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4001af4c-pmtrz\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.423531 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.423492    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4001af4c-pmtrz\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.423720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.423604    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4001af4c-pmtrz\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.423720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.423674    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvw8g\" (UniqueName: \"kubernetes.io/projected/cda96c2d-0405-4840-9836-6357f68f9b68-kube-api-access-xvw8g\") pod \"maas-api-revoke-sub-4001af4c-pmtrz\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.424251 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.424222    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4001af4c-pmtrz\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.426085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.426065    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4001af4c-pmtrz\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.481175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.481100    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvw8g\" (UniqueName: \"kubernetes.io/projected/cda96c2d-0405-4840-9836-6357f68f9b68-kube-api-access-xvw8g\") pod \"maas-api-revoke-sub-4001af4c-pmtrz\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.515797 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.515759    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"]
Oct 31 19:42:50.520678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.520624    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.555554 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:50.555523    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1406693d_432d_4f84_9d81_d0bdc062f65c.slice/crio-3142292e253330aeda4c77ddfe93f53d8d325aa88824315964b48e4a4d0d2e22 WatchSource:0}: Error finding container 3142292e253330aeda4c77ddfe93f53d8d325aa88824315964b48e4a4d0d2e22: Status 404 returned error can't find the container with id 3142292e253330aeda4c77ddfe93f53d8d325aa88824315964b48e4a4d0d2e22
Oct 31 19:42:50.558629 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.558606    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"]
Oct 31 19:42:50.563811 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.563788    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:50.574252 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.574224    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"]
Oct 31 19:42:50.625856 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.625815    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ddd4236-4f686\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.626014 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.625889    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzpf\" (UniqueName: \"kubernetes.io/projected/a9558d2b-5fbd-45bb-a667-b28220cb402d-kube-api-access-ljzpf\") pod \"maas-api-revoke-sub-1ddd4236-4f686\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.626056 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.626020    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ddd4236-4f686\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.726893 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.726858    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ddd4236-4f686\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.727085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.726908    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzpf\" (UniqueName: \"kubernetes.io/projected/a9558d2b-5fbd-45bb-a667-b28220cb402d-kube-api-access-ljzpf\") pod \"maas-api-revoke-sub-1ddd4236-4f686\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.727085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.726978    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ddd4236-4f686\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.727592 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.727565    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ddd4236-4f686\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.729232 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.729210    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ddd4236-4f686\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.757849 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.757772    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f" event={"ID":"1406693d-432d-4f84-9d81-d0bdc062f65c","Type":"ContainerStarted","Data":"3142292e253330aeda4c77ddfe93f53d8d325aa88824315964b48e4a4d0d2e22"}
Oct 31 19:42:50.784663 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.784626    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzpf\" (UniqueName: \"kubernetes.io/projected/a9558d2b-5fbd-45bb-a667-b28220cb402d-kube-api-access-ljzpf\") pod \"maas-api-revoke-sub-1ddd4236-4f686\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.834715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.834684    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:50.952249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:50.952220    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"]
Oct 31 19:42:50.953112 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:50.953086    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda96c2d_0405_4840_9836_6357f68f9b68.slice/crio-3dee7c99bebf0ae2630ba3601ee98935cc522beff879e6d469d4bcac733f7824 WatchSource:0}: Error finding container 3dee7c99bebf0ae2630ba3601ee98935cc522beff879e6d469d4bcac733f7824: Status 404 returned error can't find the container with id 3dee7c99bebf0ae2630ba3601ee98935cc522beff879e6d469d4bcac733f7824
Oct 31 19:42:51.200785 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:51.200752    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9558d2b_5fbd_45bb_a667_b28220cb402d.slice/crio-44b6f42228980e5ee6dbc80303e483e7169036c09eb8f7c3febaf02abef01baf WatchSource:0}: Error finding container 44b6f42228980e5ee6dbc80303e483e7169036c09eb8f7c3febaf02abef01baf: Status 404 returned error can't find the container with id 44b6f42228980e5ee6dbc80303e483e7169036c09eb8f7c3febaf02abef01baf
Oct 31 19:42:51.205237 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:51.205196    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"]
Oct 31 19:42:51.765581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:51.765469    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686" event={"ID":"a9558d2b-5fbd-45bb-a667-b28220cb402d","Type":"ContainerStarted","Data":"d76d80456008111cd870f42c2ca84b49ce9cb2177c3693e1d692548f71bdd459"}
Oct 31 19:42:51.765581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:51.765532    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686" event={"ID":"a9558d2b-5fbd-45bb-a667-b28220cb402d","Type":"ContainerStarted","Data":"44b6f42228980e5ee6dbc80303e483e7169036c09eb8f7c3febaf02abef01baf"}
Oct 31 19:42:51.767063 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:51.767036    5006 generic.go:356] "Generic (PLEG): container finished" podID="1406693d-432d-4f84-9d81-d0bdc062f65c" containerID="bd04325f20effc08c9b554afac01e6b51b618b4988d001ff682d28254a3ce577" exitCode=0
Oct 31 19:42:51.767165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:51.767116    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f" event={"ID":"1406693d-432d-4f84-9d81-d0bdc062f65c","Type":"ContainerDied","Data":"bd04325f20effc08c9b554afac01e6b51b618b4988d001ff682d28254a3ce577"}
Oct 31 19:42:51.768511 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:51.768490    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz" event={"ID":"cda96c2d-0405-4840-9836-6357f68f9b68","Type":"ContainerStarted","Data":"f92695fb1d6e04f5d36c370bb20f13db67988d114079fc2deae9cfe9941437ff"}
Oct 31 19:42:51.768621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:51.768524    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz" event={"ID":"cda96c2d-0405-4840-9836-6357f68f9b68","Type":"ContainerStarted","Data":"3dee7c99bebf0ae2630ba3601ee98935cc522beff879e6d469d4bcac733f7824"}
Oct 31 19:42:51.880420 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:51.880356    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686" podStartSLOduration=1.880343037 podStartE2EDuration="1.880343037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:51.879843737 +0000 UTC m=+1590.409377790" watchObservedRunningTime="2026-10-31 19:42:51.880343037 +0000 UTC m=+1590.409877126"
Oct 31 19:42:51.980843 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:51.980802    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:52.109905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:52.109837    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz" podStartSLOduration=2.109819632 podStartE2EDuration="2.109819632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:52.10798654 +0000 UTC m=+1590.637520593" watchObservedRunningTime="2026-10-31 19:42:52.109819632 +0000 UTC m=+1590.639353685"
Oct 31 19:42:52.450381 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:52.450304    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:42:52.778461 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:52.778365    5006 generic.go:356] "Generic (PLEG): container finished" podID="a9558d2b-5fbd-45bb-a667-b28220cb402d" containerID="d76d80456008111cd870f42c2ca84b49ce9cb2177c3693e1d692548f71bdd459" exitCode=0
Oct 31 19:42:52.778642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:52.778507    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686" event={"ID":"a9558d2b-5fbd-45bb-a667-b28220cb402d","Type":"ContainerDied","Data":"d76d80456008111cd870f42c2ca84b49ce9cb2177c3693e1d692548f71bdd459"}
Oct 31 19:42:52.781330 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:52.781301    5006 generic.go:356] "Generic (PLEG): container finished" podID="cda96c2d-0405-4840-9836-6357f68f9b68" containerID="f92695fb1d6e04f5d36c370bb20f13db67988d114079fc2deae9cfe9941437ff" exitCode=0
Oct 31 19:42:52.781465 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:52.781416    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz" event={"ID":"cda96c2d-0405-4840-9836-6357f68f9b68","Type":"ContainerDied","Data":"f92695fb1d6e04f5d36c370bb20f13db67988d114079fc2deae9cfe9941437ff"}
Oct 31 19:42:52.972985 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:52.972898    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:53.053320 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.053236    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-openshift-ca\") pod \"1406693d-432d-4f84-9d81-d0bdc062f65c\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") "
Oct 31 19:42:53.053721 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.053335    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1406693d-432d-4f84-9d81-d0bdc062f65c-kube-api-access-k857k\" (UniqueName: \"kubernetes.io/projected/1406693d-432d-4f84-9d81-d0bdc062f65c-kube-api-access-k857k\") pod \"1406693d-432d-4f84-9d81-d0bdc062f65c\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") "
Oct 31 19:42:53.053721 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.053371    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-xks-ca\") pod \"1406693d-432d-4f84-9d81-d0bdc062f65c\" (UID: \"1406693d-432d-4f84-9d81-d0bdc062f65c\") "
Oct 31 19:42:53.053721 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.053635    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-openshift-ca" pod "1406693d-432d-4f84-9d81-d0bdc062f65c" (UID: "1406693d-432d-4f84-9d81-d0bdc062f65c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:53.053857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.053838    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.055558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.055525    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-xks-ca" pod "1406693d-432d-4f84-9d81-d0bdc062f65c" (UID: "1406693d-432d-4f84-9d81-d0bdc062f65c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:53.055688 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.055530    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1406693d-432d-4f84-9d81-d0bdc062f65c-kube-api-access-k857k" pod "1406693d-432d-4f84-9d81-d0bdc062f65c" (UID: "1406693d-432d-4f84-9d81-d0bdc062f65c"). InnerVolumeSpecName "kube-api-access-k857k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:53.155124 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.155084    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k857k\" (UniqueName: \"kubernetes.io/projected/1406693d-432d-4f84-9d81-d0bdc062f65c-kube-api-access-k857k\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.155124 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.155118    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1406693d-432d-4f84-9d81-d0bdc062f65c-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.290009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.289961    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"]
Oct 31 19:42:53.290617 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.290594    5006 cpu_manager.go:414] "containerMap: removing container" podUID="1406693d-432d-4f84-9d81-d0bdc062f65c" containerName="revoke-keys"
Oct 31 19:42:53.290617 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.290616    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1406693d-432d-4f84-9d81-d0bdc062f65c" containerName="revoke-keys"
Oct 31 19:42:53.290786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.290732    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="1406693d-432d-4f84-9d81-d0bdc062f65c" containerName="revoke-keys"
Oct 31 19:42:53.294127 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.294104    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.349179 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.349144    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"]
Oct 31 19:42:53.357405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.357365    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad5b2cc-7k46f\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.357576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.357448    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnnrg\" (UniqueName: \"kubernetes.io/projected/4d7628c4-4ae8-4294-8d26-a6799f43c14b-kube-api-access-pnnrg\") pod \"maas-api-revoke-sub-8ad5b2cc-7k46f\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.357576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.357486    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad5b2cc-7k46f\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.458504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.458468    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad5b2cc-7k46f\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.458733 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.458518    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnnrg\" (UniqueName: \"kubernetes.io/projected/4d7628c4-4ae8-4294-8d26-a6799f43c14b-kube-api-access-pnnrg\") pod \"maas-api-revoke-sub-8ad5b2cc-7k46f\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.458733 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.458543    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad5b2cc-7k46f\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.459231 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.459209    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad5b2cc-7k46f\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.460993 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.460968    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad5b2cc-7k46f\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.516533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.516497    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnnrg\" (UniqueName: \"kubernetes.io/projected/4d7628c4-4ae8-4294-8d26-a6799f43c14b-kube-api-access-pnnrg\") pod \"maas-api-revoke-sub-8ad5b2cc-7k46f\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.611282 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.611254    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:53.789450 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.789419    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"
Oct 31 19:42:53.789689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.789460    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f" event={"ID":"1406693d-432d-4f84-9d81-d0bdc062f65c","Type":"ContainerDied","Data":"3142292e253330aeda4c77ddfe93f53d8d325aa88824315964b48e4a4d0d2e22"}
Oct 31 19:42:53.789689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.789507    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3142292e253330aeda4c77ddfe93f53d8d325aa88824315964b48e4a4d0d2e22"
Oct 31 19:42:53.940337 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.940309    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:53.973890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.973857    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:53.982359 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:53.982333    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7628c4_4ae8_4294_8d26_a6799f43c14b.slice/crio-232129acedba2f613ece64feb4a5ba6cbd69341e25f50e83adf84794e2757f90 WatchSource:0}: Error finding container 232129acedba2f613ece64feb4a5ba6cbd69341e25f50e83adf84794e2757f90: Status 404 returned error can't find the container with id 232129acedba2f613ece64feb4a5ba6cbd69341e25f50e83adf84794e2757f90
Oct 31 19:42:53.985910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:53.985881    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"]
Oct 31 19:42:54.065969 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.065934    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9558d2b-5fbd-45bb-a667-b28220cb402d-kube-api-access-ljzpf\" (UniqueName: \"kubernetes.io/projected/a9558d2b-5fbd-45bb-a667-b28220cb402d-kube-api-access-ljzpf\") pod \"a9558d2b-5fbd-45bb-a667-b28220cb402d\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") "
Oct 31 19:42:54.066470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.066019    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda96c2d-0405-4840-9836-6357f68f9b68-kube-api-access-xvw8g\" (UniqueName: \"kubernetes.io/projected/cda96c2d-0405-4840-9836-6357f68f9b68-kube-api-access-xvw8g\") pod \"cda96c2d-0405-4840-9836-6357f68f9b68\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") "
Oct 31 19:42:54.066470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.066257    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-xks-ca\") pod \"a9558d2b-5fbd-45bb-a667-b28220cb402d\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") "
Oct 31 19:42:54.066470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.066321    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-xks-ca\") pod \"cda96c2d-0405-4840-9836-6357f68f9b68\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") "
Oct 31 19:42:54.066917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.066889    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-openshift-ca\") pod \"cda96c2d-0405-4840-9836-6357f68f9b68\" (UID: \"cda96c2d-0405-4840-9836-6357f68f9b68\") "
Oct 31 19:42:54.067090 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.066970    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-openshift-ca\") pod \"a9558d2b-5fbd-45bb-a667-b28220cb402d\" (UID: \"a9558d2b-5fbd-45bb-a667-b28220cb402d\") "
Oct 31 19:42:54.067844 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.067818    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-openshift-ca" pod "cda96c2d-0405-4840-9836-6357f68f9b68" (UID: "cda96c2d-0405-4840-9836-6357f68f9b68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:54.067987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.067970    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-openshift-ca" pod "a9558d2b-5fbd-45bb-a667-b28220cb402d" (UID: "a9558d2b-5fbd-45bb-a667-b28220cb402d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:54.072064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.072035    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9558d2b-5fbd-45bb-a667-b28220cb402d-kube-api-access-ljzpf" pod "a9558d2b-5fbd-45bb-a667-b28220cb402d" (UID: "a9558d2b-5fbd-45bb-a667-b28220cb402d"). InnerVolumeSpecName "kube-api-access-ljzpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:54.073093 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.073063    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda96c2d-0405-4840-9836-6357f68f9b68-kube-api-access-xvw8g" pod "cda96c2d-0405-4840-9836-6357f68f9b68" (UID: "cda96c2d-0405-4840-9836-6357f68f9b68"). InnerVolumeSpecName "kube-api-access-xvw8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:54.073303 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.073284    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-xks-ca" pod "cda96c2d-0405-4840-9836-6357f68f9b68" (UID: "cda96c2d-0405-4840-9836-6357f68f9b68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:54.074924 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.074902    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-xks-ca" pod "a9558d2b-5fbd-45bb-a667-b28220cb402d" (UID: "a9558d2b-5fbd-45bb-a667-b28220cb402d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:54.168569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.168482    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvw8g\" (UniqueName: \"kubernetes.io/projected/cda96c2d-0405-4840-9836-6357f68f9b68-kube-api-access-xvw8g\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:54.168569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.168525    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:54.168569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.168544    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:54.168569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.168559    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda96c2d-0405-4840-9836-6357f68f9b68-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:54.168942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.168573    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9558d2b-5fbd-45bb-a667-b28220cb402d-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:54.168942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.168589    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljzpf\" (UniqueName: \"kubernetes.io/projected/a9558d2b-5fbd-45bb-a667-b28220cb402d-kube-api-access-ljzpf\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:54.797142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.797047    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686" event={"ID":"a9558d2b-5fbd-45bb-a667-b28220cb402d","Type":"ContainerDied","Data":"44b6f42228980e5ee6dbc80303e483e7169036c09eb8f7c3febaf02abef01baf"}
Oct 31 19:42:54.797142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.797074    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"
Oct 31 19:42:54.797142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.797112    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44b6f42228980e5ee6dbc80303e483e7169036c09eb8f7c3febaf02abef01baf"
Oct 31 19:42:54.798976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.798943    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f" event={"ID":"4d7628c4-4ae8-4294-8d26-a6799f43c14b","Type":"ContainerStarted","Data":"684abb653c6ac2dc973c81e4cf72abc94fc7e3ce59fa43e7de8e5abbe1f00d9d"}
Oct 31 19:42:54.799115 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.799002    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f" event={"ID":"4d7628c4-4ae8-4294-8d26-a6799f43c14b","Type":"ContainerStarted","Data":"232129acedba2f613ece64feb4a5ba6cbd69341e25f50e83adf84794e2757f90"}
Oct 31 19:42:54.800532 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.800515    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"
Oct 31 19:42:54.800686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.800521    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz" event={"ID":"cda96c2d-0405-4840-9836-6357f68f9b68","Type":"ContainerDied","Data":"3dee7c99bebf0ae2630ba3601ee98935cc522beff879e6d469d4bcac733f7824"}
Oct 31 19:42:54.800686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.800583    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dee7c99bebf0ae2630ba3601ee98935cc522beff879e6d469d4bcac733f7824"
Oct 31 19:42:54.925762 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:54.925707    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f" podStartSLOduration=1.92569039 podStartE2EDuration="1.92569039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:54.923876915 +0000 UTC m=+1593.453410969" watchObservedRunningTime="2026-10-31 19:42:54.92569039 +0000 UTC m=+1593.455224459"
Oct 31 19:42:55.356044 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.356007    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"]
Oct 31 19:42:55.356502 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.356488    5006 cpu_manager.go:414] "containerMap: removing container" podUID="a9558d2b-5fbd-45bb-a667-b28220cb402d" containerName="revoke-keys"
Oct 31 19:42:55.356548 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.356503    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9558d2b-5fbd-45bb-a667-b28220cb402d" containerName="revoke-keys"
Oct 31 19:42:55.356548 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.356541    5006 cpu_manager.go:414] "containerMap: removing container" podUID="cda96c2d-0405-4840-9836-6357f68f9b68" containerName="revoke-keys"
Oct 31 19:42:55.356548 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.356547    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda96c2d-0405-4840-9836-6357f68f9b68" containerName="revoke-keys"
Oct 31 19:42:55.356639 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.356625    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9558d2b-5fbd-45bb-a667-b28220cb402d" containerName="revoke-keys"
Oct 31 19:42:55.356688 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.356638    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda96c2d-0405-4840-9836-6357f68f9b68" containerName="revoke-keys"
Oct 31 19:42:55.359975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.359954    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.414845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.414812    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"]
Oct 31 19:42:55.482334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.482300    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrqmh\" (UniqueName: \"kubernetes.io/projected/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-kube-api-access-mrqmh\") pod \"maas-api-revoke-sub-e9d2c7f2-c5krx\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.482520 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.482501    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9d2c7f2-c5krx\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.482581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.482565    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9d2c7f2-c5krx\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.584179 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.584136    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9d2c7f2-c5krx\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.584354 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.584200    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9d2c7f2-c5krx\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.584354 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.584242    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrqmh\" (UniqueName: \"kubernetes.io/projected/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-kube-api-access-mrqmh\") pod \"maas-api-revoke-sub-e9d2c7f2-c5krx\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.585056 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.585029    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9d2c7f2-c5krx\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.586778 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.586753    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9d2c7f2-c5krx\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.641997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.641921    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrqmh\" (UniqueName: \"kubernetes.io/projected/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-kube-api-access-mrqmh\") pod \"maas-api-revoke-sub-e9d2c7f2-c5krx\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.673867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.673823    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:55.809075 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.809037    5006 generic.go:356] "Generic (PLEG): container finished" podID="4d7628c4-4ae8-4294-8d26-a6799f43c14b" containerID="684abb653c6ac2dc973c81e4cf72abc94fc7e3ce59fa43e7de8e5abbe1f00d9d" exitCode=0
Oct 31 19:42:55.809259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:55.809126    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f" event={"ID":"4d7628c4-4ae8-4294-8d26-a6799f43c14b","Type":"ContainerDied","Data":"684abb653c6ac2dc973c81e4cf72abc94fc7e3ce59fa43e7de8e5abbe1f00d9d"}
Oct 31 19:42:56.045477 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:42:56.045439    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bcc1a5_0be2_46eb_95bc_79425cb3a39e.slice/crio-50bdc87d3f0b535237db332b7d5d850a9ad41b418c31e963bab37d2ef3726daf WatchSource:0}: Error finding container 50bdc87d3f0b535237db332b7d5d850a9ad41b418c31e963bab37d2ef3726daf: Status 404 returned error can't find the container with id 50bdc87d3f0b535237db332b7d5d850a9ad41b418c31e963bab37d2ef3726daf
Oct 31 19:42:56.049382 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:56.049355    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"]
Oct 31 19:42:56.817410 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:56.817375    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx" event={"ID":"86bcc1a5-0be2-46eb-95bc-79425cb3a39e","Type":"ContainerStarted","Data":"01fb87885fb6804e8f5e0b19c0e8830a24761f780e6ac74939d8778e36b33152"}
Oct 31 19:42:56.818127 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:56.817433    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx" event={"ID":"86bcc1a5-0be2-46eb-95bc-79425cb3a39e","Type":"ContainerStarted","Data":"50bdc87d3f0b535237db332b7d5d850a9ad41b418c31e963bab37d2ef3726daf"}
Oct 31 19:42:56.936177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:56.936124    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx" podStartSLOduration=1.936108154 podStartE2EDuration="1.936108154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:56.933136221 +0000 UTC m=+1595.462670274" watchObservedRunningTime="2026-10-31 19:42:56.936108154 +0000 UTC m=+1595.465642208"
Oct 31 19:42:56.971487 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:56.971460    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:56.988727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:56.988700    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:57.101645 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.101611    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-openshift-ca\") pod \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") "
Oct 31 19:42:57.101848 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.101749    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7628c4-4ae8-4294-8d26-a6799f43c14b-kube-api-access-pnnrg\" (UniqueName: \"kubernetes.io/projected/4d7628c4-4ae8-4294-8d26-a6799f43c14b-kube-api-access-pnnrg\") pod \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") "
Oct 31 19:42:57.101848 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.101808    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-xks-ca\") pod \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\" (UID: \"4d7628c4-4ae8-4294-8d26-a6799f43c14b\") "
Oct 31 19:42:57.102359 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.102332    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-openshift-ca" pod "4d7628c4-4ae8-4294-8d26-a6799f43c14b" (UID: "4d7628c4-4ae8-4294-8d26-a6799f43c14b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:57.104894 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.104858    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-xks-ca" pod "4d7628c4-4ae8-4294-8d26-a6799f43c14b" (UID: "4d7628c4-4ae8-4294-8d26-a6799f43c14b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:57.105170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.105132    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7628c4-4ae8-4294-8d26-a6799f43c14b-kube-api-access-pnnrg" pod "4d7628c4-4ae8-4294-8d26-a6799f43c14b" (UID: "4d7628c4-4ae8-4294-8d26-a6799f43c14b"). InnerVolumeSpecName "kube-api-access-pnnrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:57.203581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.203546    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:57.203581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.203575    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7628c4-4ae8-4294-8d26-a6799f43c14b-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:57.203581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.203587    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnnrg\" (UniqueName: \"kubernetes.io/projected/4d7628c4-4ae8-4294-8d26-a6799f43c14b-kube-api-access-pnnrg\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:57.828914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.825513    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f" event={"ID":"4d7628c4-4ae8-4294-8d26-a6799f43c14b","Type":"ContainerDied","Data":"232129acedba2f613ece64feb4a5ba6cbd69341e25f50e83adf84794e2757f90"}
Oct 31 19:42:57.828914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.825574    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"
Oct 31 19:42:57.828914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.825628    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="232129acedba2f613ece64feb4a5ba6cbd69341e25f50e83adf84794e2757f90"
Oct 31 19:42:57.831032 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.830956    5006 generic.go:356] "Generic (PLEG): container finished" podID="86bcc1a5-0be2-46eb-95bc-79425cb3a39e" containerID="01fb87885fb6804e8f5e0b19c0e8830a24761f780e6ac74939d8778e36b33152" exitCode=0
Oct 31 19:42:57.831364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:57.831314    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx" event={"ID":"86bcc1a5-0be2-46eb-95bc-79425cb3a39e","Type":"ContainerDied","Data":"01fb87885fb6804e8f5e0b19c0e8830a24761f780e6ac74939d8778e36b33152"}
Oct 31 19:42:58.994665 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:58.994626    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:59.125245 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.125214    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-kube-api-access-mrqmh\" (UniqueName: \"kubernetes.io/projected/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-kube-api-access-mrqmh\") pod \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") "
Oct 31 19:42:59.125424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.125328    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-xks-ca\") pod \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") "
Oct 31 19:42:59.125424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.125419    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-openshift-ca\") pod \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\" (UID: \"86bcc1a5-0be2-46eb-95bc-79425cb3a39e\") "
Oct 31 19:42:59.125804 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.125781    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-openshift-ca" pod "86bcc1a5-0be2-46eb-95bc-79425cb3a39e" (UID: "86bcc1a5-0be2-46eb-95bc-79425cb3a39e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:59.125941 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.125919    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:59.127446 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.127420    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-kube-api-access-mrqmh" pod "86bcc1a5-0be2-46eb-95bc-79425cb3a39e" (UID: "86bcc1a5-0be2-46eb-95bc-79425cb3a39e"). InnerVolumeSpecName "kube-api-access-mrqmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:59.127825 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.127804    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-xks-ca" pod "86bcc1a5-0be2-46eb-95bc-79425cb3a39e" (UID: "86bcc1a5-0be2-46eb-95bc-79425cb3a39e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:59.226935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.226895    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrqmh\" (UniqueName: \"kubernetes.io/projected/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-kube-api-access-mrqmh\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:59.226935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.226927    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86bcc1a5-0be2-46eb-95bc-79425cb3a39e-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:59.847470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.847440    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"
Oct 31 19:42:59.847470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.847461    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx" event={"ID":"86bcc1a5-0be2-46eb-95bc-79425cb3a39e","Type":"ContainerDied","Data":"50bdc87d3f0b535237db332b7d5d850a9ad41b418c31e963bab37d2ef3726daf"}
Oct 31 19:42:59.847708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:42:59.847510    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50bdc87d3f0b535237db332b7d5d850a9ad41b418c31e963bab37d2ef3726daf"
Oct 31 19:43:00.295496 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.295466    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"]
Oct 31 19:43:00.295977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.295963    5006 cpu_manager.go:414] "containerMap: removing container" podUID="4d7628c4-4ae8-4294-8d26-a6799f43c14b" containerName="revoke-keys"
Oct 31 19:43:00.295977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.295977    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7628c4-4ae8-4294-8d26-a6799f43c14b" containerName="revoke-keys"
Oct 31 19:43:00.296047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.295997    5006 cpu_manager.go:414] "containerMap: removing container" podUID="86bcc1a5-0be2-46eb-95bc-79425cb3a39e" containerName="revoke-keys"
Oct 31 19:43:00.296047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.296002    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86bcc1a5-0be2-46eb-95bc-79425cb3a39e" containerName="revoke-keys"
Oct 31 19:43:00.296104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.296097    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7628c4-4ae8-4294-8d26-a6799f43c14b" containerName="revoke-keys"
Oct 31 19:43:00.296141 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.296106    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="86bcc1a5-0be2-46eb-95bc-79425cb3a39e" containerName="revoke-keys"
Oct 31 19:43:00.299614 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.299593    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.354399 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.354366    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"]
Oct 31 19:43:00.401011 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.400979    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:43:00.401192 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.401028    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:43:00.401192 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.401040    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:43:00.402514 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.402491    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:43:00.438862 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.438828    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f7e061-d824-460b-8030-fb6842528190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a4af8d2-jlck7\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.439012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.438870    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f7e061-d824-460b-8030-fb6842528190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a4af8d2-jlck7\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.439012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.438893    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kdsl\" (UniqueName: \"kubernetes.io/projected/27f7e061-d824-460b-8030-fb6842528190-kube-api-access-8kdsl\") pod \"maas-api-revoke-sub-8a4af8d2-jlck7\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.540182 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.540136    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f7e061-d824-460b-8030-fb6842528190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a4af8d2-jlck7\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.540182 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.540187    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f7e061-d824-460b-8030-fb6842528190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a4af8d2-jlck7\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.540443 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.540223    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kdsl\" (UniqueName: \"kubernetes.io/projected/27f7e061-d824-460b-8030-fb6842528190-kube-api-access-8kdsl\") pod \"maas-api-revoke-sub-8a4af8d2-jlck7\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.540948 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.540919    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f7e061-d824-460b-8030-fb6842528190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a4af8d2-jlck7\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.542675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.542641    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f7e061-d824-460b-8030-fb6842528190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a4af8d2-jlck7\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.597759 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.597722    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kdsl\" (UniqueName: \"kubernetes.io/projected/27f7e061-d824-460b-8030-fb6842528190-kube-api-access-8kdsl\") pod \"maas-api-revoke-sub-8a4af8d2-jlck7\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.613574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.613536    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:00.982598 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:43:00.982567    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f7e061_d824_460b_8030_fb6842528190.slice/crio-fc051dd69463ffcf3d0ecdd76b7574f8cab80825c80035ef6105365237211ad8 WatchSource:0}: Error finding container fc051dd69463ffcf3d0ecdd76b7574f8cab80825c80035ef6105365237211ad8: Status 404 returned error can't find the container with id fc051dd69463ffcf3d0ecdd76b7574f8cab80825c80035ef6105365237211ad8
Oct 31 19:43:00.987643 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:00.987619    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"]
Oct 31 19:43:01.266734 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.266644    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"]
Oct 31 19:43:01.271684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.271641    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.318730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.318694    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"]
Oct 31 19:43:01.323072 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.323045    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.333714 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.331151    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b5155c-openshift-default-dockercfg-d5fhs\""
Oct 31 19:43:01.338721 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.338693    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"]
Oct 31 19:43:01.350736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.350698    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.350890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.350751    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbjw8\" (UniqueName: \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-kube-api-access-dbjw8\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.350890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.350830    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007305fa-ae8e-4596-9dde-778b8fad4957-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.350890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.350868    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.351147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.350935    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.351147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.350980    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007305fa-ae8e-4596-9dde-778b8fad4957-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.351147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.351017    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.351147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.351077    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.351369 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.351188    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.378421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.378390    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"]
Oct 31 19:43:01.452315 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452282    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359fcc72-wz5bf\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.452508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452350    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.452508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452411    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbjw8\" (UniqueName: \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-kube-api-access-dbjw8\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.452508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452451    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007305fa-ae8e-4596-9dde-778b8fad4957-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.452508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452487    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.452508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452509    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.452820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452556    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007305fa-ae8e-4596-9dde-778b8fad4957-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.452820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452609    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.452820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452634    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.452820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452707    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssqr\" (UniqueName: \"kubernetes.io/projected/6fd15eca-c23e-4dee-a552-9295213ffa66-kube-api-access-fssqr\") pod \"maas-api-revoke-sub-359fcc72-wz5bf\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.452820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452760    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.452820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452779    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.453031 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452860    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.453031 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.452926    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359fcc72-wz5bf\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.453132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.453055    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.453315 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.453293    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.453491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.453470    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007305fa-ae8e-4596-9dde-778b8fad4957-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.455064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.455038    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.455378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.455359    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007305fa-ae8e-4596-9dde-778b8fad4957-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.514920 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.514885    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.515144 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.515121    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbjw8\" (UniqueName: \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-kube-api-access-dbjw8\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.554947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.554854    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssqr\" (UniqueName: \"kubernetes.io/projected/6fd15eca-c23e-4dee-a552-9295213ffa66-kube-api-access-fssqr\") pod \"maas-api-revoke-sub-359fcc72-wz5bf\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.555174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.555153    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359fcc72-wz5bf\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.555286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.555237    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359fcc72-wz5bf\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.555900 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.555876    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359fcc72-wz5bf\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.557391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.557372    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359fcc72-wz5bf\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.585086 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.585058    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:01.613792 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.613763    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssqr\" (UniqueName: \"kubernetes.io/projected/6fd15eca-c23e-4dee-a552-9295213ffa66-kube-api-access-fssqr\") pod \"maas-api-revoke-sub-359fcc72-wz5bf\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.645562 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.645518    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:01.806994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.806903    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:01.862541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.862504    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7" event={"ID":"27f7e061-d824-460b-8030-fb6842528190","Type":"ContainerStarted","Data":"9cb8f72a0c7a16cceb170c538ff5ae7966007717529b4305a0d91032153089de"}
Oct 31 19:43:01.862739 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.862569    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7" event={"ID":"27f7e061-d824-460b-8030-fb6842528190","Type":"ContainerStarted","Data":"fc051dd69463ffcf3d0ecdd76b7574f8cab80825c80035ef6105365237211ad8"}
Oct 31 19:43:01.981021 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:01.980962    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7" podStartSLOduration=1.9809478280000001 podStartE2EDuration="1.980947828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:01.977624148 +0000 UTC m=+1600.507158183" watchObservedRunningTime="2026-10-31 19:43:01.980947828 +0000 UTC m=+1600.510481881"
Oct 31 19:43:02.020356 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.020311    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"]
Oct 31 19:43:02.025084 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:43:02.025054    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007305fa_ae8e_4596_9dde_778b8fad4957.slice/crio-41a38127ba6411b39480ca1115cf6f593aacd6a2363577c20129f8cdc2dc100b WatchSource:0}: Error finding container 41a38127ba6411b39480ca1115cf6f593aacd6a2363577c20129f8cdc2dc100b: Status 404 returned error can't find the container with id 41a38127ba6411b39480ca1115cf6f593aacd6a2363577c20129f8cdc2dc100b
Oct 31 19:43:02.027592 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.027558    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:02.027726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.027623    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:02.027726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.027700    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:02.063588 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:43:02.063564    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd15eca_c23e_4dee_a552_9295213ffa66.slice/crio-af149f05117dad27461f8c73b2cc5cc02fac66c71fe33e7ffaad9c88bece3603 WatchSource:0}: Error finding container af149f05117dad27461f8c73b2cc5cc02fac66c71fe33e7ffaad9c88bece3603: Status 404 returned error can't find the container with id af149f05117dad27461f8c73b2cc5cc02fac66c71fe33e7ffaad9c88bece3603
Oct 31 19:43:02.067745 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.067719    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"]
Oct 31 19:43:02.870304 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.870265    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" event={"ID":"007305fa-ae8e-4596-9dde-778b8fad4957","Type":"ContainerStarted","Data":"39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5"}
Oct 31 19:43:02.870768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.870326    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" event={"ID":"007305fa-ae8e-4596-9dde-778b8fad4957","Type":"ContainerStarted","Data":"41a38127ba6411b39480ca1115cf6f593aacd6a2363577c20129f8cdc2dc100b"}
Oct 31 19:43:02.871867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.871836    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf" event={"ID":"6fd15eca-c23e-4dee-a552-9295213ffa66","Type":"ContainerStarted","Data":"8c1c6aa45b70f4efdf251dade736f5c995cf91595f0646416b46e6ea52410d98"}
Oct 31 19:43:02.872004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.871885    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf" event={"ID":"6fd15eca-c23e-4dee-a552-9295213ffa66","Type":"ContainerStarted","Data":"af149f05117dad27461f8c73b2cc5cc02fac66c71fe33e7ffaad9c88bece3603"}
Oct 31 19:43:02.873402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.873377    5006 generic.go:356] "Generic (PLEG): container finished" podID="27f7e061-d824-460b-8030-fb6842528190" containerID="9cb8f72a0c7a16cceb170c538ff5ae7966007717529b4305a0d91032153089de" exitCode=0
Oct 31 19:43:02.873501 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.873457    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7" event={"ID":"27f7e061-d824-460b-8030-fb6842528190","Type":"ContainerDied","Data":"9cb8f72a0c7a16cceb170c538ff5ae7966007717529b4305a0d91032153089de"}
Oct 31 19:43:02.992750 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:02.992692    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" podStartSLOduration=1.992678685 podStartE2EDuration="1.992678685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:02.988522728 +0000 UTC m=+1601.518056780" watchObservedRunningTime="2026-10-31 19:43:02.992678685 +0000 UTC m=+1601.522212735"
Oct 31 19:43:03.244169 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:03.244049    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf" podStartSLOduration=2.24403212 podStartE2EDuration="2.24403212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:03.243109966 +0000 UTC m=+1601.772644025" watchObservedRunningTime="2026-10-31 19:43:03.24403212 +0000 UTC m=+1601.773566174"
Oct 31 19:43:03.586730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:03.586698    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:03.588552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:03.588529    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:43:03.588672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:03.588576    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 19:43:03.881479 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:03.881392    5006 generic.go:356] "Generic (PLEG): container finished" podID="6fd15eca-c23e-4dee-a552-9295213ffa66" containerID="8c1c6aa45b70f4efdf251dade736f5c995cf91595f0646416b46e6ea52410d98" exitCode=0
Oct 31 19:43:03.881905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:03.881469    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf" event={"ID":"6fd15eca-c23e-4dee-a552-9295213ffa66","Type":"ContainerDied","Data":"8c1c6aa45b70f4efdf251dade736f5c995cf91595f0646416b46e6ea52410d98"}
Oct 31 19:43:04.079159 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.079131    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:04.154786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.154689    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:04.185135 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.185104    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f7e061-d824-460b-8030-fb6842528190-kube-api-access-8kdsl\" (UniqueName: \"kubernetes.io/projected/27f7e061-d824-460b-8030-fb6842528190-kube-api-access-8kdsl\") pod \"27f7e061-d824-460b-8030-fb6842528190\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") "
Oct 31 19:43:04.185325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.185151    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f7e061-d824-460b-8030-fb6842528190-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f7e061-d824-460b-8030-fb6842528190-maas-api-xks-ca\") pod \"27f7e061-d824-460b-8030-fb6842528190\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") "
Oct 31 19:43:04.185325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.185171    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f7e061-d824-460b-8030-fb6842528190-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f7e061-d824-460b-8030-fb6842528190-maas-api-openshift-ca\") pod \"27f7e061-d824-460b-8030-fb6842528190\" (UID: \"27f7e061-d824-460b-8030-fb6842528190\") "
Oct 31 19:43:04.185553 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.185516    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f7e061-d824-460b-8030-fb6842528190-maas-api-openshift-ca" pod "27f7e061-d824-460b-8030-fb6842528190" (UID: "27f7e061-d824-460b-8030-fb6842528190"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:04.187407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.187378    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f7e061-d824-460b-8030-fb6842528190-maas-api-xks-ca" pod "27f7e061-d824-460b-8030-fb6842528190" (UID: "27f7e061-d824-460b-8030-fb6842528190"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:04.187533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.187492    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f7e061-d824-460b-8030-fb6842528190-kube-api-access-8kdsl" pod "27f7e061-d824-460b-8030-fb6842528190" (UID: "27f7e061-d824-460b-8030-fb6842528190"). InnerVolumeSpecName "kube-api-access-8kdsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:04.286600 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.286563    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kdsl\" (UniqueName: \"kubernetes.io/projected/27f7e061-d824-460b-8030-fb6842528190-kube-api-access-8kdsl\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.286600 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.286597    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f7e061-d824-460b-8030-fb6842528190-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.286600 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.286606    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f7e061-d824-460b-8030-fb6842528190-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.586022 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.585989    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:43:04.586218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.586052    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 19:43:04.923316 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.923221    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"
Oct 31 19:43:04.923316 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.923288    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7" event={"ID":"27f7e061-d824-460b-8030-fb6842528190","Type":"ContainerDied","Data":"fc051dd69463ffcf3d0ecdd76b7574f8cab80825c80035ef6105365237211ad8"}
Oct 31 19:43:04.923853 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:04.923355    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc051dd69463ffcf3d0ecdd76b7574f8cab80825c80035ef6105365237211ad8"
Oct 31 19:43:05.094615 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.094582    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:05.223763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.221998    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-openshift-ca\") pod \"6fd15eca-c23e-4dee-a552-9295213ffa66\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") "
Oct 31 19:43:05.223763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.222152    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd15eca-c23e-4dee-a552-9295213ffa66-kube-api-access-fssqr\" (UniqueName: \"kubernetes.io/projected/6fd15eca-c23e-4dee-a552-9295213ffa66-kube-api-access-fssqr\") pod \"6fd15eca-c23e-4dee-a552-9295213ffa66\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") "
Oct 31 19:43:05.223763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.222232    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-xks-ca\") pod \"6fd15eca-c23e-4dee-a552-9295213ffa66\" (UID: \"6fd15eca-c23e-4dee-a552-9295213ffa66\") "
Oct 31 19:43:05.225294 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.225261    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-openshift-ca" pod "6fd15eca-c23e-4dee-a552-9295213ffa66" (UID: "6fd15eca-c23e-4dee-a552-9295213ffa66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:05.227869 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.227843    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-xks-ca" pod "6fd15eca-c23e-4dee-a552-9295213ffa66" (UID: "6fd15eca-c23e-4dee-a552-9295213ffa66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:05.228310 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.228284    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd15eca-c23e-4dee-a552-9295213ffa66-kube-api-access-fssqr" pod "6fd15eca-c23e-4dee-a552-9295213ffa66" (UID: "6fd15eca-c23e-4dee-a552-9295213ffa66"). InnerVolumeSpecName "kube-api-access-fssqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:05.324494 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.324449    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssqr\" (UniqueName: \"kubernetes.io/projected/6fd15eca-c23e-4dee-a552-9295213ffa66-kube-api-access-fssqr\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:05.324494 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.324496    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:05.324763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.324512    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd15eca-c23e-4dee-a552-9295213ffa66-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:05.393472 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:43:05.393435    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:43:05.587170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.587136    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:43:05.587324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.587189    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 19:43:05.934599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.934522    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"
Oct 31 19:43:05.934599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.934560    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf" event={"ID":"6fd15eca-c23e-4dee-a552-9295213ffa66","Type":"ContainerDied","Data":"af149f05117dad27461f8c73b2cc5cc02fac66c71fe33e7ffaad9c88bece3603"}
Oct 31 19:43:05.935045 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:05.934611    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af149f05117dad27461f8c73b2cc5cc02fac66c71fe33e7ffaad9c88bece3603"
Oct 31 19:43:06.586386 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.586350    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:43:06.586561 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.586411    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 19:43:06.871936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.871849    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"]
Oct 31 19:43:06.872340 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.872327    5006 cpu_manager.go:414] "containerMap: removing container" podUID="6fd15eca-c23e-4dee-a552-9295213ffa66" containerName="revoke-keys"
Oct 31 19:43:06.872379 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.872342    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd15eca-c23e-4dee-a552-9295213ffa66" containerName="revoke-keys"
Oct 31 19:43:06.872379 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.872363    5006 cpu_manager.go:414] "containerMap: removing container" podUID="27f7e061-d824-460b-8030-fb6842528190" containerName="revoke-keys"
Oct 31 19:43:06.872379 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.872369    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f7e061-d824-460b-8030-fb6842528190" containerName="revoke-keys"
Oct 31 19:43:06.872475 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.872453    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f7e061-d824-460b-8030-fb6842528190" containerName="revoke-keys"
Oct 31 19:43:06.872475 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.872467    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd15eca-c23e-4dee-a552-9295213ffa66" containerName="revoke-keys"
Oct 31 19:43:06.876104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.876084    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:06.930566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.930528    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"]
Oct 31 19:43:06.977150 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.977114    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:43:06.977560 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.977163    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:43:06.977560 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.977224    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:43:06.978988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:06.978971    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:43:07.042914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.042886    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c6a600a-j69d2\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:07.043097 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.042922    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cc98\" (UniqueName: \"kubernetes.io/projected/fae6c4b7-98d7-4345-9c19-c42d18c90729-kube-api-access-9cc98\") pod \"maas-api-revoke-sub-3c6a600a-j69d2\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:07.043144 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.043101    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c6a600a-j69d2\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:07.144332 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.144225    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c6a600a-j69d2\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:07.144332 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.144287    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cc98\" (UniqueName: \"kubernetes.io/projected/fae6c4b7-98d7-4345-9c19-c42d18c90729-kube-api-access-9cc98\") pod \"maas-api-revoke-sub-3c6a600a-j69d2\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:07.144569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.144446    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c6a600a-j69d2\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:07.145120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.145088    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c6a600a-j69d2\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:07.146627 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.146604    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c6a600a-j69d2\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:07.221580 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.221481    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cc98\" (UniqueName: \"kubernetes.io/projected/fae6c4b7-98d7-4345-9c19-c42d18c90729-kube-api-access-9cc98\") pod \"maas-api-revoke-sub-3c6a600a-j69d2\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:07.464981 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.464893    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:07.489591 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.489559    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:07.594153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.592073    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:43:07.594153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.592134    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 19:43:07.887875 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:43:07.887842    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae6c4b7_98d7_4345_9c19_c42d18c90729.slice/crio-596690a950c738fec172eca542cff3194ed5594fc647ba66760c963b6033c159 WatchSource:0}: Error finding container 596690a950c738fec172eca542cff3194ed5594fc647ba66760c963b6033c159: Status 404 returned error can't find the container with id 596690a950c738fec172eca542cff3194ed5594fc647ba66760c963b6033c159
Oct 31 19:43:07.890959 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.890935    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"]
Oct 31 19:43:07.952708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:07.952676    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2" event={"ID":"fae6c4b7-98d7-4345-9c19-c42d18c90729","Type":"ContainerStarted","Data":"596690a950c738fec172eca542cff3194ed5594fc647ba66760c963b6033c159"}
Oct 31 19:43:08.256266 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:08.256227    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:08.586418 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:08.586386    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:43:08.586578 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:08.586452    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 19:43:08.960499 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:08.960398    5006 generic.go:356] "Generic (PLEG): container finished" podID="fae6c4b7-98d7-4345-9c19-c42d18c90729" containerID="20303049ddfc157f8af2565e0d093ace991fcff4a61ef0103e4be7129fced1cd" exitCode=0
Oct 31 19:43:08.960499 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:08.960449    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2" event={"ID":"fae6c4b7-98d7-4345-9c19-c42d18c90729","Type":"ContainerDied","Data":"20303049ddfc157f8af2565e0d093ace991fcff4a61ef0103e4be7129fced1cd"}
Oct 31 19:43:09.093789 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:09.093748    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:09.319007 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:09.318919    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:09.598457 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:09.598432    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:09.598870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:09.598852    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:09.599894 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:09.599872    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:43:10.124145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.124123    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:10.284390 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.284296    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-openshift-ca\") pod \"fae6c4b7-98d7-4345-9c19-c42d18c90729\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") "
Oct 31 19:43:10.284390 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.284372    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-xks-ca\") pod \"fae6c4b7-98d7-4345-9c19-c42d18c90729\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") "
Oct 31 19:43:10.284630 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.284479    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae6c4b7-98d7-4345-9c19-c42d18c90729-kube-api-access-9cc98\" (UniqueName: \"kubernetes.io/projected/fae6c4b7-98d7-4345-9c19-c42d18c90729-kube-api-access-9cc98\") pod \"fae6c4b7-98d7-4345-9c19-c42d18c90729\" (UID: \"fae6c4b7-98d7-4345-9c19-c42d18c90729\") "
Oct 31 19:43:10.284787 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.284760    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-openshift-ca" pod "fae6c4b7-98d7-4345-9c19-c42d18c90729" (UID: "fae6c4b7-98d7-4345-9c19-c42d18c90729"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:10.284962 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.284945    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:10.286834 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.286805    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae6c4b7-98d7-4345-9c19-c42d18c90729-kube-api-access-9cc98" pod "fae6c4b7-98d7-4345-9c19-c42d18c90729" (UID: "fae6c4b7-98d7-4345-9c19-c42d18c90729"). InnerVolumeSpecName "kube-api-access-9cc98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:10.287056 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.287029    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-xks-ca" pod "fae6c4b7-98d7-4345-9c19-c42d18c90729" (UID: "fae6c4b7-98d7-4345-9c19-c42d18c90729"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:10.386758 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.386715    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae6c4b7-98d7-4345-9c19-c42d18c90729-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:10.386758 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.386758    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cc98\" (UniqueName: \"kubernetes.io/projected/fae6c4b7-98d7-4345-9c19-c42d18c90729-kube-api-access-9cc98\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:10.974908 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.974872    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"
Oct 31 19:43:10.975096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.974902    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2" event={"ID":"fae6c4b7-98d7-4345-9c19-c42d18c90729","Type":"ContainerDied","Data":"596690a950c738fec172eca542cff3194ed5594fc647ba66760c963b6033c159"}
Oct 31 19:43:10.975096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:10.974960    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="596690a950c738fec172eca542cff3194ed5594fc647ba66760c963b6033c159"
Oct 31 19:43:12.164368 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.164335    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"]
Oct 31 19:43:12.165218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.165188    5006 cpu_manager.go:414] "containerMap: removing container" podUID="fae6c4b7-98d7-4345-9c19-c42d18c90729" containerName="revoke-keys"
Oct 31 19:43:12.165218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.165218    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae6c4b7-98d7-4345-9c19-c42d18c90729" containerName="revoke-keys"
Oct 31 19:43:12.165429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.165360    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae6c4b7-98d7-4345-9c19-c42d18c90729" containerName="revoke-keys"
Oct 31 19:43:12.169445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.169423    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.205457 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.205427    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d787d57c-lcv9z\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.205721 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.205466    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h56hm\" (UniqueName: \"kubernetes.io/projected/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-kube-api-access-h56hm\") pod \"maas-api-revoke-sub-d787d57c-lcv9z\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.205721 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.205491    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d787d57c-lcv9z\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.214186 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.214153    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"]
Oct 31 19:43:12.260541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.260483    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:43:12.260541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.260535    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:43:12.260838 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.260613    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:43:12.272546 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.272523    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:43:12.306629 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.306598    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d787d57c-lcv9z\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.306799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.306640    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h56hm\" (UniqueName: \"kubernetes.io/projected/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-kube-api-access-h56hm\") pod \"maas-api-revoke-sub-d787d57c-lcv9z\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.306799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.306684    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d787d57c-lcv9z\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.307298 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.307270    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d787d57c-lcv9z\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.308972 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.308950    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d787d57c-lcv9z\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.367623 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.367587    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h56hm\" (UniqueName: \"kubernetes.io/projected/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-kube-api-access-h56hm\") pod \"maas-api-revoke-sub-d787d57c-lcv9z\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.488828 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.488737    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:12.512505 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.512455    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:12.872591 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.872563    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"]
Oct 31 19:43:12.987862 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:12.987827    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z" event={"ID":"1e380ee2-2c4c-4529-aba0-f5de7368c1f3","Type":"ContainerStarted","Data":"8ec8b6596549c2a63f6e6769faf68cd79b99f890f5c324ba390b6c9df80d6807"}
Oct 31 19:43:13.997251 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:13.997216    5006 generic.go:356] "Generic (PLEG): container finished" podID="1e380ee2-2c4c-4529-aba0-f5de7368c1f3" containerID="4e8af9200ad772cc25d27fff3661e73c2134924caa9cb45dfc5d8af764822a3d" exitCode=0
Oct 31 19:43:13.997704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:13.997301    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z" event={"ID":"1e380ee2-2c4c-4529-aba0-f5de7368c1f3","Type":"ContainerDied","Data":"4e8af9200ad772cc25d27fff3661e73c2134924caa9cb45dfc5d8af764822a3d"}
Oct 31 19:43:15.182345 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:15.182309    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:15.235406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:15.235367    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-kube-api-access-h56hm\" (UniqueName: \"kubernetes.io/projected/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-kube-api-access-h56hm\") pod \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") "
Oct 31 19:43:15.235586 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:15.235462    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-openshift-ca\") pod \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") "
Oct 31 19:43:15.235688 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:15.235589    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-xks-ca\") pod \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\" (UID: \"1e380ee2-2c4c-4529-aba0-f5de7368c1f3\") "
Oct 31 19:43:15.236009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:15.235924    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-openshift-ca" pod "1e380ee2-2c4c-4529-aba0-f5de7368c1f3" (UID: "1e380ee2-2c4c-4529-aba0-f5de7368c1f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:15.236175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:15.236148    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:15.238125 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:15.238095    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-kube-api-access-h56hm" pod "1e380ee2-2c4c-4529-aba0-f5de7368c1f3" (UID: "1e380ee2-2c4c-4529-aba0-f5de7368c1f3"). InnerVolumeSpecName "kube-api-access-h56hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:15.238702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:15.238676    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-xks-ca" pod "1e380ee2-2c4c-4529-aba0-f5de7368c1f3" (UID: "1e380ee2-2c4c-4529-aba0-f5de7368c1f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:15.337092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:15.337052    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h56hm\" (UniqueName: \"kubernetes.io/projected/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-kube-api-access-h56hm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:15.337251 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:15.337096    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e380ee2-2c4c-4529-aba0-f5de7368c1f3-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:16.014162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.014081    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"
Oct 31 19:43:16.014162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.014119    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z" event={"ID":"1e380ee2-2c4c-4529-aba0-f5de7368c1f3","Type":"ContainerDied","Data":"8ec8b6596549c2a63f6e6769faf68cd79b99f890f5c324ba390b6c9df80d6807"}
Oct 31 19:43:16.014347 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.014176    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ec8b6596549c2a63f6e6769faf68cd79b99f890f5c324ba390b6c9df80d6807"
Oct 31 19:43:16.134147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.134104    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"]
Oct 31 19:43:16.134744 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.134723    5006 cpu_manager.go:414] "containerMap: removing container" podUID="1e380ee2-2c4c-4529-aba0-f5de7368c1f3" containerName="revoke-keys"
Oct 31 19:43:16.134864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.134744    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e380ee2-2c4c-4529-aba0-f5de7368c1f3" containerName="revoke-keys"
Oct 31 19:43:16.134928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.134878    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e380ee2-2c4c-4529-aba0-f5de7368c1f3" containerName="revoke-keys"
Oct 31 19:43:16.138443 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.138422    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.191750 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.191714    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5a4438-openshift-default-dockercfg-7kvr6\""
Oct 31 19:43:16.248082 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.248041    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"]
Oct 31 19:43:16.248728 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.248692    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.248869 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.248740    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66g9\" (UniqueName: \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-kube-api-access-z66g9\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.248932 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.248861    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.248932 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.248911    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.249039 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.248995    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.249039 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.249027    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.249148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.249077    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.249148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.249101    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.249254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.249165    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.349960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.349918    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350025    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350049    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66g9\" (UniqueName: \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-kube-api-access-z66g9\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350096    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350134    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350214    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350246    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350301    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350332    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350361    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350620 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350427    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350620 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350467    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350707 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350684    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.350995 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.350974    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.352627 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.352601    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.352965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.352945    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.407937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.407901    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.408090 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.407967    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66g9\" (UniqueName: \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-kube-api-access-z66g9\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.451827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.451794    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:16.871317 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:43:16.871278    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd4ccb3_3226_42c4_a0d9_a6d7ac564125.slice/crio-df758f99e28c5422defbddbbd2838c6f9debda5aa365c7ae291ac3f52c10e104 WatchSource:0}: Error finding container df758f99e28c5422defbddbbd2838c6f9debda5aa365c7ae291ac3f52c10e104: Status 404 returned error can't find the container with id df758f99e28c5422defbddbbd2838c6f9debda5aa365c7ae291ac3f52c10e104
Oct 31 19:43:16.873941 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.873902    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:16.874033 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.873993    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:16.874033 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.874025    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:16.874730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:16.874701    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"]
Oct 31 19:43:17.022840 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.022802    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" event={"ID":"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125","Type":"ContainerStarted","Data":"2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b"}
Oct 31 19:43:17.023029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.022863    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" event={"ID":"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125","Type":"ContainerStarted","Data":"df758f99e28c5422defbddbbd2838c6f9debda5aa365c7ae291ac3f52c10e104"}
Oct 31 19:43:17.153752 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.153617    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" podStartSLOduration=1.153603116 podStartE2EDuration="1.153603116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:17.152602845 +0000 UTC m=+1615.682136899" watchObservedRunningTime="2026-10-31 19:43:17.153603116 +0000 UTC m=+1615.683137169"
Oct 31 19:43:17.452562 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.452471    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:17.454453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.454424    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:43:17.454586 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.454480    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 19:43:17.530827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.530789    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"]
Oct 31 19:43:17.535776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.535748    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.565946 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.565904    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wbnq\" (UniqueName: \"kubernetes.io/projected/fbb9545d-adef-490b-974f-b28ebb9076ac-kube-api-access-5wbnq\") pod \"maas-api-revoke-sub-d5ba2503-x6lqh\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.566147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.566031    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ba2503-x6lqh\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.566204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.566146    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ba2503-x6lqh\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.593593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.593558    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:43:17.593593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.593558    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:43:17.593593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.593559    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:43:17.593908 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.593629    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:43:17.600071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.600038    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"]
Oct 31 19:43:17.648355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.648305    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"]
Oct 31 19:43:17.654337 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.654299    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.667055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.667020    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ba2503-x6lqh\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.667484 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.667454    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wbnq\" (UniqueName: \"kubernetes.io/projected/fbb9545d-adef-490b-974f-b28ebb9076ac-kube-api-access-5wbnq\") pod \"maas-api-revoke-sub-d5ba2503-x6lqh\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.667723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.667694    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ba2503-x6lqh\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.669038 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.669014    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ba2503-x6lqh\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.670598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.670574    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ba2503-x6lqh\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.714737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.712608    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-662be1-openshift-default-dockercfg-bwkr4\""
Oct 31 19:43:17.721674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.719386    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"]
Oct 31 19:43:17.730039 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.730000    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wbnq\" (UniqueName: \"kubernetes.io/projected/fbb9545d-adef-490b-974f-b28ebb9076ac-kube-api-access-5wbnq\") pod \"maas-api-revoke-sub-d5ba2503-x6lqh\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.741160 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.741121    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"]
Oct 31 19:43:17.746553 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.746525    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:17.776137 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776100    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27c55924-rqbgs\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:17.776318 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776160    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.776318 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776274    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.776318 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776303    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27c55924-rqbgs\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:17.776488 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776330    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.776488 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776470    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tth4k\" (UniqueName: \"kubernetes.io/projected/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-kube-api-access-tth4k\") pod \"maas-api-revoke-sub-27c55924-rqbgs\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:17.776585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776507    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdrv\" (UniqueName: \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-kube-api-access-5rdrv\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.776585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776557    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.776705 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776626    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc592dbb-a347-4476-a4c3-17987158ffa2-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.776761 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776728    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.776815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776796    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.776864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.776835    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.828205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.828159    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"]
Oct 31 19:43:17.841782 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.841745    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:17.851257 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.851223    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:17.878635 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.878594    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc592dbb-a347-4476-a4c3-17987158ffa2-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.878825 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.878716    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.878825 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.878785    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.878947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.878825    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.878947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.878882    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27c55924-rqbgs\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:17.878947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.878917    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.879092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.878990    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.879092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.879020    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27c55924-rqbgs\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:17.879092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.879055    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.879250 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.879117    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tth4k\" (UniqueName: \"kubernetes.io/projected/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-kube-api-access-tth4k\") pod \"maas-api-revoke-sub-27c55924-rqbgs\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:17.879250 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.879142    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdrv\" (UniqueName: \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-kube-api-access-5rdrv\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.879250 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.879190    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.879929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.879487    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc592dbb-a347-4476-a4c3-17987158ffa2-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.879929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.879607    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.879929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.879852    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.880115 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.879950    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.880368 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.880344    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.880483 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.880421    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27c55924-rqbgs\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:17.882388 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.882355    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27c55924-rqbgs\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:17.882613 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.882596    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.882897 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.882878    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.937543 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.937511    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdrv\" (UniqueName: \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-kube-api-access-5rdrv\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.940294 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.938178    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:17.940294 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.939224    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tth4k\" (UniqueName: \"kubernetes.io/projected/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-kube-api-access-tth4k\") pod \"maas-api-revoke-sub-27c55924-rqbgs\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:17.973417 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:17.973321    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:18.079187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:18.079033    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:18.255293 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:43:18.254963    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb9545d_adef_490b_974f_b28ebb9076ac.slice/crio-923fe2fd5321a6b9b214c37a8f2c35fc6ce135a5cf1f5c559869deaab6c7bc63 WatchSource:0}: Error finding container 923fe2fd5321a6b9b214c37a8f2c35fc6ce135a5cf1f5c559869deaab6c7bc63: Status 404 returned error can't find the container with id 923fe2fd5321a6b9b214c37a8f2c35fc6ce135a5cf1f5c559869deaab6c7bc63
Oct 31 19:43:18.256770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:18.256744    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"]
Oct 31 19:43:18.389399 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:43:18.389365    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc592dbb_a347_4476_a4c3_17987158ffa2.slice/crio-4da872d131c6c358a1b1d1eb8d44f92748555f52e9a0a5f400bc55a1253bcdd8 WatchSource:0}: Error finding container 4da872d131c6c358a1b1d1eb8d44f92748555f52e9a0a5f400bc55a1253bcdd8: Status 404 returned error can't find the container with id 4da872d131c6c358a1b1d1eb8d44f92748555f52e9a0a5f400bc55a1253bcdd8
Oct 31 19:43:18.392120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:18.392080    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:18.392241 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:18.392153    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:18.392241 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:18.392181    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:18.393335 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:18.393307    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"]
Oct 31 19:43:18.452953 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:18.452923    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:43:18.453371 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:18.452979    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 19:43:18.484145 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:43:18.484113    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc26859a_5db3_49c9_a1b7_68ff3b07ed11.slice/crio-19556601c3c7912cd758e3a8c02d4f664ce320fbdac82b4f9cb84589c3f0a662 WatchSource:0}: Error finding container 19556601c3c7912cd758e3a8c02d4f664ce320fbdac82b4f9cb84589c3f0a662: Status 404 returned error can't find the container with id 19556601c3c7912cd758e3a8c02d4f664ce320fbdac82b4f9cb84589c3f0a662
Oct 31 19:43:18.487661 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:18.487623    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"]
Oct 31 19:43:19.040582 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.040543    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs" event={"ID":"dc26859a-5db3-49c9-a1b7-68ff3b07ed11","Type":"ContainerStarted","Data":"df2e4bb5edb981b9c99cda24beb6a01f8dc97545a4fd3c828bbc0ad7647b391c"}
Oct 31 19:43:19.040780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.040612    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs" event={"ID":"dc26859a-5db3-49c9-a1b7-68ff3b07ed11","Type":"ContainerStarted","Data":"19556601c3c7912cd758e3a8c02d4f664ce320fbdac82b4f9cb84589c3f0a662"}
Oct 31 19:43:19.042329 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.042299    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh" event={"ID":"fbb9545d-adef-490b-974f-b28ebb9076ac","Type":"ContainerStarted","Data":"4ecefcab230e1ac623d098ae46748c22a563108eadcf50f6094fde143aaf8620"}
Oct 31 19:43:19.042453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.042345    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh" event={"ID":"fbb9545d-adef-490b-974f-b28ebb9076ac","Type":"ContainerStarted","Data":"923fe2fd5321a6b9b214c37a8f2c35fc6ce135a5cf1f5c559869deaab6c7bc63"}
Oct 31 19:43:19.043977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.043952    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" event={"ID":"fc592dbb-a347-4476-a4c3-17987158ffa2","Type":"ContainerStarted","Data":"d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4"}
Oct 31 19:43:19.044064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.043995    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" event={"ID":"fc592dbb-a347-4476-a4c3-17987158ffa2","Type":"ContainerStarted","Data":"4da872d131c6c358a1b1d1eb8d44f92748555f52e9a0a5f400bc55a1253bcdd8"}
Oct 31 19:43:19.294171 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.294058    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs" podStartSLOduration=2.294042874 podStartE2EDuration="2.294042874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:19.166258906 +0000 UTC m=+1617.695792958" watchObservedRunningTime="2026-10-31 19:43:19.294042874 +0000 UTC m=+1617.823576964"
Oct 31 19:43:19.295641 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.295606    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh" podStartSLOduration=2.295597269 podStartE2EDuration="2.295597269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:19.291085983 +0000 UTC m=+1617.820620035" watchObservedRunningTime="2026-10-31 19:43:19.295597269 +0000 UTC m=+1617.825131322"
Oct 31 19:43:19.447844 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.447501    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" podStartSLOduration=2.447479276 podStartE2EDuration="2.447479276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:19.435577031 +0000 UTC m=+1617.965111125" watchObservedRunningTime="2026-10-31 19:43:19.447479276 +0000 UTC m=+1617.977013329"
Oct 31 19:43:19.454002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.453829    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:43:19.454002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.453889    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 19:43:19.973955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.973918    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:19.975688 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.975439    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:43:19.975688 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:19.975496    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 19:43:20.057044 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:20.056945    5006 generic.go:356] "Generic (PLEG): container finished" podID="dc26859a-5db3-49c9-a1b7-68ff3b07ed11" containerID="df2e4bb5edb981b9c99cda24beb6a01f8dc97545a4fd3c828bbc0ad7647b391c" exitCode=0
Oct 31 19:43:20.057225 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:20.057037    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs" event={"ID":"dc26859a-5db3-49c9-a1b7-68ff3b07ed11","Type":"ContainerDied","Data":"df2e4bb5edb981b9c99cda24beb6a01f8dc97545a4fd3c828bbc0ad7647b391c"}
Oct 31 19:43:20.058760 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:20.058734    5006 generic.go:356] "Generic (PLEG): container finished" podID="fbb9545d-adef-490b-974f-b28ebb9076ac" containerID="4ecefcab230e1ac623d098ae46748c22a563108eadcf50f6094fde143aaf8620" exitCode=0
Oct 31 19:43:20.058884 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:20.058800    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh" event={"ID":"fbb9545d-adef-490b-974f-b28ebb9076ac","Type":"ContainerDied","Data":"4ecefcab230e1ac623d098ae46748c22a563108eadcf50f6094fde143aaf8620"}
Oct 31 19:43:20.453085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:20.453052    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:43:20.453302 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:20.453108    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 19:43:20.572508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:20.572471    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:20.974043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:20.974009    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:43:20.974278 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:20.974255    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 19:43:21.381541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.381517    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:21.453223 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.453179    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:43:21.454598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.454565    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 19:43:21.457047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.457015    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:21.473381 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.473349    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-openshift-ca\") pod \"fbb9545d-adef-490b-974f-b28ebb9076ac\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") "
Oct 31 19:43:21.473563 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.473468    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-xks-ca\") pod \"fbb9545d-adef-490b-974f-b28ebb9076ac\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") "
Oct 31 19:43:21.473563 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.473525    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb9545d-adef-490b-974f-b28ebb9076ac-kube-api-access-5wbnq\" (UniqueName: \"kubernetes.io/projected/fbb9545d-adef-490b-974f-b28ebb9076ac-kube-api-access-5wbnq\") pod \"fbb9545d-adef-490b-974f-b28ebb9076ac\" (UID: \"fbb9545d-adef-490b-974f-b28ebb9076ac\") "
Oct 31 19:43:21.473739 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.473711    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-openshift-ca" pod "fbb9545d-adef-490b-974f-b28ebb9076ac" (UID: "fbb9545d-adef-490b-974f-b28ebb9076ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:21.473998 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.473981    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:21.475459 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.475441    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-xks-ca" pod "fbb9545d-adef-490b-974f-b28ebb9076ac" (UID: "fbb9545d-adef-490b-974f-b28ebb9076ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:21.475620 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.475607    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb9545d-adef-490b-974f-b28ebb9076ac-kube-api-access-5wbnq" pod "fbb9545d-adef-490b-974f-b28ebb9076ac" (UID: "fbb9545d-adef-490b-974f-b28ebb9076ac"). InnerVolumeSpecName "kube-api-access-5wbnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:21.575338 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.575221    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-xks-ca\") pod \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") "
Oct 31 19:43:21.575338 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.575309    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-openshift-ca\") pod \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") "
Oct 31 19:43:21.575937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.575398    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-kube-api-access-tth4k\" (UniqueName: \"kubernetes.io/projected/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-kube-api-access-tth4k\") pod \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\" (UID: \"dc26859a-5db3-49c9-a1b7-68ff3b07ed11\") "
Oct 31 19:43:21.575937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.575926    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb9545d-adef-490b-974f-b28ebb9076ac-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:21.576058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.575954    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wbnq\" (UniqueName: \"kubernetes.io/projected/fbb9545d-adef-490b-974f-b28ebb9076ac-kube-api-access-5wbnq\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:21.576690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.576630    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-openshift-ca" pod "dc26859a-5db3-49c9-a1b7-68ff3b07ed11" (UID: "dc26859a-5db3-49c9-a1b7-68ff3b07ed11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:21.579379 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.579338    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-kube-api-access-tth4k" pod "dc26859a-5db3-49c9-a1b7-68ff3b07ed11" (UID: "dc26859a-5db3-49c9-a1b7-68ff3b07ed11"). InnerVolumeSpecName "kube-api-access-tth4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:21.579691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.579639    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-xks-ca" pod "dc26859a-5db3-49c9-a1b7-68ff3b07ed11" (UID: "dc26859a-5db3-49c9-a1b7-68ff3b07ed11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:21.678364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.677794    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:21.678364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.677856    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:21.678364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.677886    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tth4k\" (UniqueName: \"kubernetes.io/projected/dc26859a-5db3-49c9-a1b7-68ff3b07ed11-kube-api-access-tth4k\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:21.974275 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.974232    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:43:21.974440 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:21.974290    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 19:43:22.074595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.074554    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs" event={"ID":"dc26859a-5db3-49c9-a1b7-68ff3b07ed11","Type":"ContainerDied","Data":"19556601c3c7912cd758e3a8c02d4f664ce320fbdac82b4f9cb84589c3f0a662"}
Oct 31 19:43:22.074772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.074639    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19556601c3c7912cd758e3a8c02d4f664ce320fbdac82b4f9cb84589c3f0a662"
Oct 31 19:43:22.074772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.074562    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"
Oct 31 19:43:22.076044 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.076021    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"
Oct 31 19:43:22.076130 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.076046    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh" event={"ID":"fbb9545d-adef-490b-974f-b28ebb9076ac","Type":"ContainerDied","Data":"923fe2fd5321a6b9b214c37a8f2c35fc6ce135a5cf1f5c559869deaab6c7bc63"}
Oct 31 19:43:22.076130 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.076083    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="923fe2fd5321a6b9b214c37a8f2c35fc6ce135a5cf1f5c559869deaab6c7bc63"
Oct 31 19:43:22.453171 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.453125    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:43:22.453360 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.453195    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 19:43:22.830042 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.829996    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:22.974917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.974871    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:43:22.975108 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:22.974937    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 19:43:23.452601 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:23.452565    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:43:23.452806 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:23.452629    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 19:43:23.973989 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:23.973943    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:43:23.974498 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:23.974001    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 19:43:24.488607 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:24.488574    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:24.489058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:24.489029    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:24.490031 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:24.490010    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 19:43:24.974689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:24.974629    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:43:24.975119 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:24.974713    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 19:43:25.978941 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:25.978911    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:25.979461 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:25.979300    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:25.980057 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:25.980031    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 19:43:27.425030 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.424988    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"]
Oct 31 19:43:27.425819 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.425792    5006 cpu_manager.go:414] "containerMap: removing container" podUID="dc26859a-5db3-49c9-a1b7-68ff3b07ed11" containerName="revoke-keys"
Oct 31 19:43:27.425819 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.425818    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc26859a-5db3-49c9-a1b7-68ff3b07ed11" containerName="revoke-keys"
Oct 31 19:43:27.426003 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.425848    5006 cpu_manager.go:414] "containerMap: removing container" podUID="fbb9545d-adef-490b-974f-b28ebb9076ac" containerName="revoke-keys"
Oct 31 19:43:27.426003 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.425857    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb9545d-adef-490b-974f-b28ebb9076ac" containerName="revoke-keys"
Oct 31 19:43:27.426104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.426000    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc26859a-5db3-49c9-a1b7-68ff3b07ed11" containerName="revoke-keys"
Oct 31 19:43:27.426104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.426017    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb9545d-adef-490b-974f-b28ebb9076ac" containerName="revoke-keys"
Oct 31 19:43:27.430935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.430910    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.459416 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.459377    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a1a0bbb-f4mmt\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.459593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.459489    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a1a0bbb-f4mmt\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.459593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.459547    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgbwd\" (UniqueName: \"kubernetes.io/projected/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-kube-api-access-qgbwd\") pod \"maas-api-revoke-sub-4a1a0bbb-f4mmt\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.484782 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.484743    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"]
Oct 31 19:43:27.530942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.530904    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:43:27.530942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.530906    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:43:27.530942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.530920    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:43:27.540017 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.539984    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:43:27.560582 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.560534    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgbwd\" (UniqueName: \"kubernetes.io/projected/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-kube-api-access-qgbwd\") pod \"maas-api-revoke-sub-4a1a0bbb-f4mmt\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.560801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.560709    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a1a0bbb-f4mmt\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.560872 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.560801    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a1a0bbb-f4mmt\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.561440 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.561414    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a1a0bbb-f4mmt\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.563205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.563186    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a1a0bbb-f4mmt\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.618325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.618291    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgbwd\" (UniqueName: \"kubernetes.io/projected/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-kube-api-access-qgbwd\") pod \"maas-api-revoke-sub-4a1a0bbb-f4mmt\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.745400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.745308    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:27.806486 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:27.806447    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:28.134181 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:43:28.134142    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ccf36a_305b_4926_a5d4_5ba5aa59f5fd.slice/crio-b4cacaa4ba57b67c67fef894c3973e6a9056fd3dc816945a048a2b85f06d9646 WatchSource:0}: Error finding container b4cacaa4ba57b67c67fef894c3973e6a9056fd3dc816945a048a2b85f06d9646: Status 404 returned error can't find the container with id b4cacaa4ba57b67c67fef894c3973e6a9056fd3dc816945a048a2b85f06d9646
Oct 31 19:43:28.136758 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:28.136737    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"]
Oct 31 19:43:29.126106 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:29.126069    5006 generic.go:356] "Generic (PLEG): container finished" podID="61ccf36a-305b-4926-a5d4-5ba5aa59f5fd" containerID="efd64f44a93424704d2070f9ceb9e7c1413c0ffa11256b5a4fddd21e23e9ca11" exitCode=0
Oct 31 19:43:29.126495 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:29.126156    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt" event={"ID":"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd","Type":"ContainerDied","Data":"efd64f44a93424704d2070f9ceb9e7c1413c0ffa11256b5a4fddd21e23e9ca11"}
Oct 31 19:43:29.126495 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:29.126204    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt" event={"ID":"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd","Type":"ContainerStarted","Data":"b4cacaa4ba57b67c67fef894c3973e6a9056fd3dc816945a048a2b85f06d9646"}
Oct 31 19:43:30.307854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:30.307821    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:30.391828 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:30.391740    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-openshift-ca\") pod \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") "
Oct 31 19:43:30.391828 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:30.391829    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-kube-api-access-qgbwd\" (UniqueName: \"kubernetes.io/projected/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-kube-api-access-qgbwd\") pod \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") "
Oct 31 19:43:30.392022 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:30.391920    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-xks-ca\") pod \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\" (UID: \"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd\") "
Oct 31 19:43:30.392129 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:30.392102    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-openshift-ca" pod "61ccf36a-305b-4926-a5d4-5ba5aa59f5fd" (UID: "61ccf36a-305b-4926-a5d4-5ba5aa59f5fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:30.392215 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:30.392201    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:30.393950 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:30.393923    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-xks-ca" pod "61ccf36a-305b-4926-a5d4-5ba5aa59f5fd" (UID: "61ccf36a-305b-4926-a5d4-5ba5aa59f5fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:30.394057 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:30.394030    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-kube-api-access-qgbwd" pod "61ccf36a-305b-4926-a5d4-5ba5aa59f5fd" (UID: "61ccf36a-305b-4926-a5d4-5ba5aa59f5fd"). InnerVolumeSpecName "kube-api-access-qgbwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:30.493456 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:30.493425    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgbwd\" (UniqueName: \"kubernetes.io/projected/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-kube-api-access-qgbwd\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:30.493456 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:30.493452    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:31.141420 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:31.141385    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"
Oct 31 19:43:31.141420 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:31.141404    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt" event={"ID":"61ccf36a-305b-4926-a5d4-5ba5aa59f5fd","Type":"ContainerDied","Data":"b4cacaa4ba57b67c67fef894c3973e6a9056fd3dc816945a048a2b85f06d9646"}
Oct 31 19:43:31.141643 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:31.141455    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4cacaa4ba57b67c67fef894c3973e6a9056fd3dc816945a048a2b85f06d9646"
Oct 31 19:43:32.729664 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:32.729621    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:35.073449 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:35.073409    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:43:35.394395 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:43:35.394350    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:43:37.711624 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:37.711590    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:42.718971 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:42.718939    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:47.692174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:47.692141    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:52.629953 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:52.629916    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:54.773966 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:54.773934    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:57.591195 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:43:57.591152    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:00.367336 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:00.367293    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:02.593610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:02.593574    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:05.393060 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:44:05.393020    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:44:05.872966 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:05.872924    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:07.587921 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:07.587878    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:07.845420 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:07.845324    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:44:12.567605 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:12.567563    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:12.837836 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:12.837794    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:17.474659 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:17.474608    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:20.901901 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:20.901839    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:22.396599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:22.396560    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:27.357695 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:27.357659    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:32.261101 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:32.261053    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:33.174272 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.174235    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"]
Oct 31 19:44:33.174898 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.174878    5006 cpu_manager.go:414] "containerMap: removing container" podUID="61ccf36a-305b-4926-a5d4-5ba5aa59f5fd" containerName="revoke-keys"
Oct 31 19:44:33.175012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.174899    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ccf36a-305b-4926-a5d4-5ba5aa59f5fd" containerName="revoke-keys"
Oct 31 19:44:33.175095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.175081    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ccf36a-305b-4926-a5d4-5ba5aa59f5fd" containerName="revoke-keys"
Oct 31 19:44:33.178563 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.178542    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.211599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.211550    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wktgc\" (UniqueName: \"kubernetes.io/projected/9c78a101-f486-45ac-a45b-e4ba2366fe68-kube-api-access-wktgc\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.211810 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.211608    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-home\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.211810 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.211745    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-model-cache\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.211927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.211900    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c78a101-f486-45ac-a45b-e4ba2366fe68-tls-certs\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.211977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.211965    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-dshm\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.212038 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.212024    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-tmp-dir\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.231716 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.231679    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"]
Oct 31 19:44:33.278280 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.278240    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-093b97\"/\"distinct-w0-093b97-kserve-self-signed-certs\""
Oct 31 19:44:33.313116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.313076    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktgc\" (UniqueName: \"kubernetes.io/projected/9c78a101-f486-45ac-a45b-e4ba2366fe68-kube-api-access-wktgc\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.313842 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.313779    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-home\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.315407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.315381    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-home\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.315601 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.315578    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-model-cache\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.315718 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.315698    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c78a101-f486-45ac-a45b-e4ba2366fe68-tls-certs\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.315868 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.315761    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-dshm\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.315868 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.315803    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-tmp-dir\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.316147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.315943    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-model-cache\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.316274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.316251    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-tmp-dir\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.318641 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.318613    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-dshm\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.319983 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.319950    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c78a101-f486-45ac-a45b-e4ba2366fe68-tls-certs\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.353956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.353925    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:33.371269 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.371242    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktgc\" (UniqueName: \"kubernetes.io/projected/9c78a101-f486-45ac-a45b-e4ba2366fe68-kube-api-access-wktgc\") pod \"distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.491811 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.491697    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:33.880092 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:44:33.880063    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c78a101_f486_45ac_a45b_e4ba2366fe68.slice/crio-55f5dfed265a2539fb58c2c3ab9b843a5ea02448852a0c1646c89acc237e7591 WatchSource:0}: Error finding container 55f5dfed265a2539fb58c2c3ab9b843a5ea02448852a0c1646c89acc237e7591: Status 404 returned error can't find the container with id 55f5dfed265a2539fb58c2c3ab9b843a5ea02448852a0c1646c89acc237e7591
Oct 31 19:44:33.884044 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:33.884021    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"]
Oct 31 19:44:34.589611 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:34.589571    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" event={"ID":"9c78a101-f486-45ac-a45b-e4ba2366fe68","Type":"ContainerStarted","Data":"e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b"}
Oct 31 19:44:34.590035 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:34.589623    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" event={"ID":"9c78a101-f486-45ac-a45b-e4ba2366fe68","Type":"ContainerStarted","Data":"55f5dfed265a2539fb58c2c3ab9b843a5ea02448852a0c1646c89acc237e7591"}
Oct 31 19:44:34.706198 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:34.706132    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podStartSLOduration=1.7061161390000001 podStartE2EDuration="1.706116139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:34.704571882 +0000 UTC m=+1693.234105946" watchObservedRunningTime="2026-10-31 19:44:34.706116139 +0000 UTC m=+1693.235650190"
Oct 31 19:44:35.394192 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:44:35.394155    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:44:35.492161 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:35.492127    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:36.328262 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:36.328212    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:37.167289 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:37.167216    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:42.057835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:42.057805    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:43.491854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:43.491822    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:43.504791 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:43.504759    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:43.671098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:43.671068    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:44:44.358356 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:44.358318    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:44:46.981717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:46.981678    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:48.371849 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.371813    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"]
Oct 31 19:44:48.376095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.376077    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.469867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.469837    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"]
Oct 31 19:44:48.501118 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.501089    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.501324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.501156    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.501324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.501256    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d0d17d-7f38-4238-87b9-733e2ab64858-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.501324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.501300    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.501324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.501319    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhpwz\" (UniqueName: \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-kube-api-access-dhpwz\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.501462 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.501390    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.501462 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.501413    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.501462 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.501433    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.501564 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.501483    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.602532 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.602498    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.602770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.602565    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.602770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.602617    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d0d17d-7f38-4238-87b9-733e2ab64858-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.602770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.602672    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.602770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.602701    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhpwz\" (UniqueName: \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-kube-api-access-dhpwz\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.602770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.602736    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.602770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.602762    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.603113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.602889    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.603113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.602937    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.603113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.602974    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.603113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.603083    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.603338 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.603275    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.603395 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.603338    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.603437 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.603394    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d0d17d-7f38-4238-87b9-733e2ab64858-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.604904 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.604883    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.605269 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.605253    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.661009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.660936    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.661142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.661043    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhpwz\" (UniqueName: \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-kube-api-access-dhpwz\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:48.690818 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:48.690791    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:49.108791 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:44:49.108754    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d0d17d_7f38_4238_87b9_733e2ab64858.slice/crio-10fdbef159ccdb1763fdc5b5179d4860eb8b06edea1ca1111e686d93ddb41218 WatchSource:0}: Error finding container 10fdbef159ccdb1763fdc5b5179d4860eb8b06edea1ca1111e686d93ddb41218: Status 404 returned error can't find the container with id 10fdbef159ccdb1763fdc5b5179d4860eb8b06edea1ca1111e686d93ddb41218
Oct 31 19:44:49.111396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:49.111362    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:44:49.111475 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:49.111429    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:44:49.111475 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:49.111461    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:44:49.113028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:49.113002    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"]
Oct 31 19:44:49.702448 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:49.702411    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" event={"ID":"36d0d17d-7f38-4238-87b9-733e2ab64858","Type":"ContainerStarted","Data":"e671837b2c04cf30b182f4db65aac2c5abc0bbdd4d60158291ff044276a72ea4"}
Oct 31 19:44:49.702910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:49.702473    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" event={"ID":"36d0d17d-7f38-4238-87b9-733e2ab64858","Type":"ContainerStarted","Data":"10fdbef159ccdb1763fdc5b5179d4860eb8b06edea1ca1111e686d93ddb41218"}
Oct 31 19:44:49.821391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:49.821336    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" podStartSLOduration=1.821317877 podStartE2EDuration="1.821317877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:49.81962031 +0000 UTC m=+1708.349154365" watchObservedRunningTime="2026-10-31 19:44:49.821317877 +0000 UTC m=+1708.350851936"
Oct 31 19:44:50.691983 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:50.691951    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:50.694140 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:50.693792    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 19:44:50.694140 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:50.693839    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 19:44:51.691528 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:51.691496    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 19:44:51.691937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:51.691549    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 19:44:52.029733 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:52.029603    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:52.691740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:52.691703    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 19:44:52.692249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:52.691769    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 19:44:53.692071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:53.692032    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 19:44:53.692471 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:53.692086    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 19:44:54.692121 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:54.692081    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 19:44:54.692500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:54.692151    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 19:44:55.337989 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.337947    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"]
Oct 31 19:44:55.342403 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.342383    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.388757 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.388717    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-tmp-dir\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.388941 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.388788    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-dshm\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.388941 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.388899    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-home\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.388941 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.388933    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zj7\" (UniqueName: \"kubernetes.io/projected/7ee071cd-147d-4bd0-aece-a823c2e7e911-kube-api-access-z8zj7\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.389109 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.389073    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-model-cache\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.389165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.389119    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee071cd-147d-4bd0-aece-a823c2e7e911-tls-certs\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.400729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.400693    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"]
Oct 31 19:44:55.443216 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.443178    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a4438\"/\"distinct-2-w6-5a4438-kserve-self-signed-certs\""
Oct 31 19:44:55.490359 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.490322    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-model-cache\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.490535 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.490366    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee071cd-147d-4bd0-aece-a823c2e7e911-tls-certs\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.490587 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.490534    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-tmp-dir\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.490642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.490591    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-dshm\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.490642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.490633    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-home\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.490787 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.490673    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zj7\" (UniqueName: \"kubernetes.io/projected/7ee071cd-147d-4bd0-aece-a823c2e7e911-kube-api-access-z8zj7\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.490934 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.490903    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-model-cache\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.491068 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.490956    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-tmp-dir\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.491068 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.490997    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-home\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.492812 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.492784    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-dshm\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.493121 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.493101    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee071cd-147d-4bd0-aece-a823c2e7e911-tls-certs\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.549770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.549732    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zj7\" (UniqueName: \"kubernetes.io/projected/7ee071cd-147d-4bd0-aece-a823c2e7e911-kube-api-access-z8zj7\") pod \"distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.560141 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.560074    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:55.655948 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.655862    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:44:55.704852 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.704382    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 19:44:55.704852 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:55.704458    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 19:44:56.049816 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:44:56.049774    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ee071cd_147d_4bd0_aece_a823c2e7e911.slice/crio-c694b7059569dddc3e8aa47ddb4d08c9784cde7f03cfe04d018df00f39e9354c WatchSource:0}: Error finding container c694b7059569dddc3e8aa47ddb4d08c9784cde7f03cfe04d018df00f39e9354c: Status 404 returned error can't find the container with id c694b7059569dddc3e8aa47ddb4d08c9784cde7f03cfe04d018df00f39e9354c
Oct 31 19:44:56.053977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:56.053952    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"]
Oct 31 19:44:56.696373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:56.696342    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:56.696787 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:56.696759    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:56.697473 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:56.697455    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:44:56.774686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:56.774617    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" event={"ID":"7ee071cd-147d-4bd0-aece-a823c2e7e911","Type":"ContainerStarted","Data":"6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451"}
Oct 31 19:44:56.774686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:56.774690    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" event={"ID":"7ee071cd-147d-4bd0-aece-a823c2e7e911","Type":"ContainerStarted","Data":"c694b7059569dddc3e8aa47ddb4d08c9784cde7f03cfe04d018df00f39e9354c"}
Oct 31 19:44:57.000075 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:56.999968    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podStartSLOduration=1.999952298 podStartE2EDuration="1.999952298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:56.997707653 +0000 UTC m=+1715.527241728" watchObservedRunningTime="2026-10-31 19:44:56.999952298 +0000 UTC m=+1715.529486353"
Oct 31 19:44:57.115691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:57.115638    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:57.657014 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:44:57.656977    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:45:00.169339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.169298    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"]
Oct 31 19:45:00.173578 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.173556    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"]
Oct 31 19:45:00.173723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.173698    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"
Oct 31 19:45:00.178114 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.178090    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"
Oct 31 19:45:00.181436 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.181418    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"]
Oct 31 19:45:00.185217 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.185198    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"
Oct 31 19:45:00.185418 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.185399    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"]
Oct 31 19:45:00.189114 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.189082    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"
Oct 31 19:45:00.219764 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.219737    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"]
Oct 31 19:45:00.223630 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.223612    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"
Oct 31 19:45:00.226039 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.226016    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:45:00.226504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.226488    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:45:00.228475 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.228458    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:45:00.268979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.268942    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"]
Oct 31 19:45:00.272519 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.272487    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"]
Oct 31 19:45:00.274968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.274941    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg\" (UID: \"69263e6c-ab4a-469e-ad65-101eb2bbe9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"
Oct 31 19:45:00.275081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.274980    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwnt\" (UniqueName: \"kubernetes.io/projected/06c132c4-683a-4baa-9919-6aa73708457e-kube-api-access-qrwnt\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95\" (UID: \"06c132c4-683a-4baa-9919-6aa73708457e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"
Oct 31 19:45:00.275161 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.275135    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pf24\" (UniqueName: \"kubernetes.io/projected/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-kube-api-access-6pf24\") pod \"maas-api-key-cleanup-29891265-dvjlc\" (UID: \"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"
Oct 31 19:45:00.275227 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.275210    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e152e6-c6f7-4b7e-b262-8eb60682024d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t\" (UID: \"88e152e6-c6f7-4b7e-b262-8eb60682024d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"
Oct 31 19:45:00.275286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.275270    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c132c4-683a-4baa-9919-6aa73708457e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95\" (UID: \"06c132c4-683a-4baa-9919-6aa73708457e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"
Oct 31 19:45:00.275403 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.275387    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5gs\" (UniqueName: \"kubernetes.io/projected/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-kube-api-access-ft5gs\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg\" (UID: \"69263e6c-ab4a-469e-ad65-101eb2bbe9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"
Oct 31 19:45:00.275464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.275418    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gslsl\" (UniqueName: \"kubernetes.io/projected/88e152e6-c6f7-4b7e-b262-8eb60682024d-kube-api-access-gslsl\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t\" (UID: \"88e152e6-c6f7-4b7e-b262-8eb60682024d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"
Oct 31 19:45:00.275508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.275456    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891265-dvjlc\" (UID: \"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"
Oct 31 19:45:00.308817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.308783    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"]
Oct 31 19:45:00.313689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.313631    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"]
Oct 31 19:45:00.371552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.371505    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"]
Oct 31 19:45:00.377235 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.377205    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg\" (UID: \"69263e6c-ab4a-469e-ad65-101eb2bbe9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"
Oct 31 19:45:00.377411 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.377241    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwnt\" (UniqueName: \"kubernetes.io/projected/06c132c4-683a-4baa-9919-6aa73708457e-kube-api-access-qrwnt\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95\" (UID: \"06c132c4-683a-4baa-9919-6aa73708457e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"
Oct 31 19:45:00.377411 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.377293    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pf24\" (UniqueName: \"kubernetes.io/projected/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-kube-api-access-6pf24\") pod \"maas-api-key-cleanup-29891265-dvjlc\" (UID: \"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"
Oct 31 19:45:00.377411 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.377328    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e152e6-c6f7-4b7e-b262-8eb60682024d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t\" (UID: \"88e152e6-c6f7-4b7e-b262-8eb60682024d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"
Oct 31 19:45:00.377411 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.377358    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c132c4-683a-4baa-9919-6aa73708457e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95\" (UID: \"06c132c4-683a-4baa-9919-6aa73708457e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"
Oct 31 19:45:00.377683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.377419    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c630ed21-9b1b-44e6-b548-89eefc9a0acf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz\" (UID: \"c630ed21-9b1b-44e6-b548-89eefc9a0acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"
Oct 31 19:45:00.377683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.377479    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5gs\" (UniqueName: \"kubernetes.io/projected/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-kube-api-access-ft5gs\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg\" (UID: \"69263e6c-ab4a-469e-ad65-101eb2bbe9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"
Oct 31 19:45:00.377683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.377504    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gslsl\" (UniqueName: \"kubernetes.io/projected/88e152e6-c6f7-4b7e-b262-8eb60682024d-kube-api-access-gslsl\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t\" (UID: \"88e152e6-c6f7-4b7e-b262-8eb60682024d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"
Oct 31 19:45:00.377683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.377549    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891265-dvjlc\" (UID: \"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"
Oct 31 19:45:00.377683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.377597    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4dw\" (UniqueName: \"kubernetes.io/projected/c630ed21-9b1b-44e6-b548-89eefc9a0acf-kube-api-access-xk4dw\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz\" (UID: \"c630ed21-9b1b-44e6-b548-89eefc9a0acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"
Oct 31 19:45:00.378138 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.378117    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg\" (UID: \"69263e6c-ab4a-469e-ad65-101eb2bbe9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"
Oct 31 19:45:00.378237 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.378214    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e152e6-c6f7-4b7e-b262-8eb60682024d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t\" (UID: \"88e152e6-c6f7-4b7e-b262-8eb60682024d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"
Oct 31 19:45:00.378279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.378256    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c132c4-683a-4baa-9919-6aa73708457e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95\" (UID: \"06c132c4-683a-4baa-9919-6aa73708457e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"
Oct 31 19:45:00.378722 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.378704    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891265-dvjlc\" (UID: \"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"
Oct 31 19:45:00.435944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.435864    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gslsl\" (UniqueName: \"kubernetes.io/projected/88e152e6-c6f7-4b7e-b262-8eb60682024d-kube-api-access-gslsl\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t\" (UID: \"88e152e6-c6f7-4b7e-b262-8eb60682024d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"
Oct 31 19:45:00.435944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.435873    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwnt\" (UniqueName: \"kubernetes.io/projected/06c132c4-683a-4baa-9919-6aa73708457e-kube-api-access-qrwnt\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95\" (UID: \"06c132c4-683a-4baa-9919-6aa73708457e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"
Oct 31 19:45:00.436111 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.436078    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5gs\" (UniqueName: \"kubernetes.io/projected/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-kube-api-access-ft5gs\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg\" (UID: \"69263e6c-ab4a-469e-ad65-101eb2bbe9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"
Oct 31 19:45:00.436172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.436157    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pf24\" (UniqueName: \"kubernetes.io/projected/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-kube-api-access-6pf24\") pod \"maas-api-key-cleanup-29891265-dvjlc\" (UID: \"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"
Oct 31 19:45:00.478901 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.478863    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c630ed21-9b1b-44e6-b548-89eefc9a0acf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz\" (UID: \"c630ed21-9b1b-44e6-b548-89eefc9a0acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"
Oct 31 19:45:00.479094 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.478941    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4dw\" (UniqueName: \"kubernetes.io/projected/c630ed21-9b1b-44e6-b548-89eefc9a0acf-kube-api-access-xk4dw\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz\" (UID: \"c630ed21-9b1b-44e6-b548-89eefc9a0acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"
Oct 31 19:45:00.479489 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.479467    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c630ed21-9b1b-44e6-b548-89eefc9a0acf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz\" (UID: \"c630ed21-9b1b-44e6-b548-89eefc9a0acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"
Oct 31 19:45:00.491202 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.491174    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"
Oct 31 19:45:00.496782 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.496753    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"
Oct 31 19:45:00.504285 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.504259    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"
Oct 31 19:45:00.511046 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.511018    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"
Oct 31 19:45:00.537375 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.537339    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4dw\" (UniqueName: \"kubernetes.io/projected/c630ed21-9b1b-44e6-b548-89eefc9a0acf-kube-api-access-xk4dw\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz\" (UID: \"c630ed21-9b1b-44e6-b548-89eefc9a0acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"
Oct 31 19:45:00.837165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.837129    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"
Oct 31 19:45:00.884013 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:45:00.883972    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c132c4_683a_4baa_9919_6aa73708457e.slice/crio-9038f81c979ba0eceebaef4178247eb93e90c29a3fe29f385bfd38a804ebea97 WatchSource:0}: Error finding container 9038f81c979ba0eceebaef4178247eb93e90c29a3fe29f385bfd38a804ebea97: Status 404 returned error can't find the container with id 9038f81c979ba0eceebaef4178247eb93e90c29a3fe29f385bfd38a804ebea97
Oct 31 19:45:00.886442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.886383    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"]
Oct 31 19:45:00.933218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.933193    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"]
Oct 31 19:45:00.934080 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:45:00.934052    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69263e6c_ab4a_469e_ad65_101eb2bbe9c6.slice/crio-b8d5519ee3f29ebd8419f9187d6e1bc5b32d7719e4755eb4977c2a0ae23a9fc8 WatchSource:0}: Error finding container b8d5519ee3f29ebd8419f9187d6e1bc5b32d7719e4755eb4977c2a0ae23a9fc8: Status 404 returned error can't find the container with id b8d5519ee3f29ebd8419f9187d6e1bc5b32d7719e4755eb4977c2a0ae23a9fc8
Oct 31 19:45:00.938725 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.938701    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"]
Oct 31 19:45:00.943383 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:00.943352    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"]
Oct 31 19:45:01.215794 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:45:01.215761    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc630ed21_9b1b_44e6_b548_89eefc9a0acf.slice/crio-d785a1aa85014ce6171ab502d563b8183ed98c1e08b7c0464d593af4b436e1ca WatchSource:0}: Error finding container d785a1aa85014ce6171ab502d563b8183ed98c1e08b7c0464d593af4b436e1ca: Status 404 returned error can't find the container with id d785a1aa85014ce6171ab502d563b8183ed98c1e08b7c0464d593af4b436e1ca
Oct 31 19:45:01.217927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.217902    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"]
Oct 31 19:45:01.814775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.814742    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" event={"ID":"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a","Type":"ContainerStarted","Data":"26d511c5a66b06500e3510cf0fe5f66e9ceb98083528f8ca872858313e3e4f43"}
Oct 31 19:45:01.814978 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.814805    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" event={"ID":"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a","Type":"ContainerStarted","Data":"b15f7ce82096d03c0182bd25b5f6a443b84ddc574b690dbea5cfc67aee73b2ff"}
Oct 31 19:45:01.816298 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.816270    5006 generic.go:356] "Generic (PLEG): container finished" podID="06c132c4-683a-4baa-9919-6aa73708457e" containerID="e4887bc42ed0ce68cfc12d6c0965f6a77c3028119e49d47e2ae86c255cb47a59" exitCode=60
Oct 31 19:45:01.816434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.816354    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95" event={"ID":"06c132c4-683a-4baa-9919-6aa73708457e","Type":"ContainerDied","Data":"e4887bc42ed0ce68cfc12d6c0965f6a77c3028119e49d47e2ae86c255cb47a59"}
Oct 31 19:45:01.816434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.816404    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95" event={"ID":"06c132c4-683a-4baa-9919-6aa73708457e","Type":"ContainerStarted","Data":"9038f81c979ba0eceebaef4178247eb93e90c29a3fe29f385bfd38a804ebea97"}
Oct 31 19:45:01.816588 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.816571    5006 scope.go:122] "RemoveContainer" containerID="e4887bc42ed0ce68cfc12d6c0965f6a77c3028119e49d47e2ae86c255cb47a59"
Oct 31 19:45:01.818096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.818026    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" event={"ID":"88e152e6-c6f7-4b7e-b262-8eb60682024d","Type":"ContainerStarted","Data":"ed6deccc03898ad3be95854abbd0fb38e7d9e2f0be55b45ef4c2f88474f3304d"}
Oct 31 19:45:01.818096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.818075    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" event={"ID":"88e152e6-c6f7-4b7e-b262-8eb60682024d","Type":"ContainerStarted","Data":"dedee12d923421771b50a6624624656a42efbacee9667bdf9a365db322efaeee"}
Oct 31 19:45:01.819766 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.819749    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" event={"ID":"c630ed21-9b1b-44e6-b548-89eefc9a0acf","Type":"ContainerStarted","Data":"239e28a2bda400416b2565e8b035211bdd92cdc69b25f38b6c3e17e114ffc0e9"}
Oct 31 19:45:01.819836 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.819794    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" event={"ID":"c630ed21-9b1b-44e6-b548-89eefc9a0acf","Type":"ContainerStarted","Data":"d785a1aa85014ce6171ab502d563b8183ed98c1e08b7c0464d593af4b436e1ca"}
Oct 31 19:45:01.821244 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.821223    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" event={"ID":"69263e6c-ab4a-469e-ad65-101eb2bbe9c6","Type":"ContainerStarted","Data":"5f8d39beaed5675df4b6a78bef96ddb960daa5768f08f32f1b30d3b90d8944cc"}
Oct 31 19:45:01.821324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.821258    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" event={"ID":"69263e6c-ab4a-469e-ad65-101eb2bbe9c6","Type":"ContainerStarted","Data":"b8d5519ee3f29ebd8419f9187d6e1bc5b32d7719e4755eb4977c2a0ae23a9fc8"}
Oct 31 19:45:01.952713 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.952625    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" podStartSLOduration=1.952605493 podStartE2EDuration="1.952605493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:01.947724677 +0000 UTC m=+1720.477258730" watchObservedRunningTime="2026-10-31 19:45:01.952605493 +0000 UTC m=+1720.482139547"
Oct 31 19:45:01.972364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:01.972273    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:02.172767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.172712    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" podStartSLOduration=2.172696124 podStartE2EDuration="2.172696124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:02.170198435 +0000 UTC m=+1720.699732507" watchObservedRunningTime="2026-10-31 19:45:02.172696124 +0000 UTC m=+1720.702230185"
Oct 31 19:45:02.284236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.284124    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" podStartSLOduration=2.284108885 podStartE2EDuration="2.284108885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:02.282127512 +0000 UTC m=+1720.811661565" watchObservedRunningTime="2026-10-31 19:45:02.284108885 +0000 UTC m=+1720.813642974"
Oct 31 19:45:02.397331 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.397278    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" podStartSLOduration=2.397262134 podStartE2EDuration="2.397262134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:02.393346247 +0000 UTC m=+1720.922880318" watchObservedRunningTime="2026-10-31 19:45:02.397262134 +0000 UTC m=+1720.926796197"
Oct 31 19:45:02.801957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.801923    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"]
Oct 31 19:45:02.805947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.805926    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:02.829861 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.829825    5006 generic.go:356] "Generic (PLEG): container finished" podID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerID="26d511c5a66b06500e3510cf0fe5f66e9ceb98083528f8ca872858313e3e4f43" exitCode=60
Oct 31 19:45:02.830040 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.829907    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" event={"ID":"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a","Type":"ContainerDied","Data":"26d511c5a66b06500e3510cf0fe5f66e9ceb98083528f8ca872858313e3e4f43"}
Oct 31 19:45:02.830172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.830155    5006 scope.go:122] "RemoveContainer" containerID="26d511c5a66b06500e3510cf0fe5f66e9ceb98083528f8ca872858313e3e4f43"
Oct 31 19:45:02.832079 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.832058    5006 generic.go:356] "Generic (PLEG): container finished" podID="06c132c4-683a-4baa-9919-6aa73708457e" containerID="703b275eadbbb42f5bd78cfc1fac36f6c21bfaec29ace0ea553c52d730a0fcb9" exitCode=60
Oct 31 19:45:02.832187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.832142    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95" event={"ID":"06c132c4-683a-4baa-9919-6aa73708457e","Type":"ContainerDied","Data":"703b275eadbbb42f5bd78cfc1fac36f6c21bfaec29ace0ea553c52d730a0fcb9"}
Oct 31 19:45:02.832259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.832195    5006 scope.go:122] "RemoveContainer" containerID="e4887bc42ed0ce68cfc12d6c0965f6a77c3028119e49d47e2ae86c255cb47a59"
Oct 31 19:45:02.832401 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.832387    5006 scope.go:122] "RemoveContainer" containerID="703b275eadbbb42f5bd78cfc1fac36f6c21bfaec29ace0ea553c52d730a0fcb9"
Oct 31 19:45:02.833207 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:02.833182    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95_odh-ai-gateway-infra(06c132c4-683a-4baa-9919-6aa73708457e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95" podUID="06c132c4-683a-4baa-9919-6aa73708457e"
Oct 31 19:45:02.834144 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.834119    5006 generic.go:356] "Generic (PLEG): container finished" podID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerID="ed6deccc03898ad3be95854abbd0fb38e7d9e2f0be55b45ef4c2f88474f3304d" exitCode=60
Oct 31 19:45:02.834248 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.834158    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" event={"ID":"88e152e6-c6f7-4b7e-b262-8eb60682024d","Type":"ContainerDied","Data":"ed6deccc03898ad3be95854abbd0fb38e7d9e2f0be55b45ef4c2f88474f3304d"}
Oct 31 19:45:02.834423 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.834409    5006 scope.go:122] "RemoveContainer" containerID="ed6deccc03898ad3be95854abbd0fb38e7d9e2f0be55b45ef4c2f88474f3304d"
Oct 31 19:45:02.835821 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.835795    5006 generic.go:356] "Generic (PLEG): container finished" podID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerID="239e28a2bda400416b2565e8b035211bdd92cdc69b25f38b6c3e17e114ffc0e9" exitCode=60
Oct 31 19:45:02.835909 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.835877    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" event={"ID":"c630ed21-9b1b-44e6-b548-89eefc9a0acf","Type":"ContainerDied","Data":"239e28a2bda400416b2565e8b035211bdd92cdc69b25f38b6c3e17e114ffc0e9"}
Oct 31 19:45:02.836081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.836063    5006 scope.go:122] "RemoveContainer" containerID="239e28a2bda400416b2565e8b035211bdd92cdc69b25f38b6c3e17e114ffc0e9"
Oct 31 19:45:02.838614 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.838586    5006 generic.go:356] "Generic (PLEG): container finished" podID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerID="5f8d39beaed5675df4b6a78bef96ddb960daa5768f08f32f1b30d3b90d8944cc" exitCode=60
Oct 31 19:45:02.838743 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.838632    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" event={"ID":"69263e6c-ab4a-469e-ad65-101eb2bbe9c6","Type":"ContainerDied","Data":"5f8d39beaed5675df4b6a78bef96ddb960daa5768f08f32f1b30d3b90d8944cc"}
Oct 31 19:45:02.838956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.838940    5006 scope.go:122] "RemoveContainer" containerID="5f8d39beaed5675df4b6a78bef96ddb960daa5768f08f32f1b30d3b90d8944cc"
Oct 31 19:45:02.858712 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.858691    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-093b97-openshift-default-dockercfg-mndxw\""
Oct 31 19:45:02.907805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.907770    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"]
Oct 31 19:45:02.912274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.912239    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z86h\" (UniqueName: \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-kube-api-access-5z86h\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:02.912434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.912284    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:02.912434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.912387    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:02.912434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.912415    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:02.912604 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.912437    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:02.912604 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.912596    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:02.912731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.912622    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:02.912731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.912640    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:02.912839 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:02.912741    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11e81ed-de32-43bf-89d7-3503c50fe558-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.013729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.013616    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.013729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.013699    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.013729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.013719    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.013982 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.013906    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11e81ed-de32-43bf-89d7-3503c50fe558-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.014110 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.014089    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z86h\" (UniqueName: \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-kube-api-access-5z86h\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.014110 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.014097    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.014240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.014126    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.014240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.014176    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.014240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.014205    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.014405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.014269    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.014405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.014299    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.014405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.014336    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.014859 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.014831    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.014964 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.014836    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11e81ed-de32-43bf-89d7-3503c50fe558-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.016472 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.016445    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.016996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.016971    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.071174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.071141    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.071325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.071199    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z86h\" (UniqueName: \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-kube-api-access-5z86h\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.132415 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.132384    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"]
Oct 31 19:45:03.137974 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.137948    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.191449 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.191401    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-967dee-openshift-default-dockercfg-btwn6\""
Oct 31 19:45:03.194354 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.194329    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"]
Oct 31 19:45:03.211018 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.210988    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:03.327351 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.327320    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447ea240-4362-41e5-b077-4daed43f4c41-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.327820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.327372    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhn8f\" (UniqueName: \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-kube-api-access-mhn8f\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.327820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.327505    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.327820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.327546    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.327820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.327573    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447ea240-4362-41e5-b077-4daed43f4c41-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.327820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.327638    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.327820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.327729    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.327820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.327755    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.327820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.327794    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.428974 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.428935    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.428997    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429027    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429064    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429181    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447ea240-4362-41e5-b077-4daed43f4c41-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429223    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhn8f\" (UniqueName: \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-kube-api-access-mhn8f\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429305    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429336    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429357    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447ea240-4362-41e5-b077-4daed43f4c41-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429445    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429461    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429520    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429784 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429693    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.429929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.429906    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447ea240-4362-41e5-b077-4daed43f4c41-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.431767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.431735    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.431850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.431824    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447ea240-4362-41e5-b077-4daed43f4c41-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.487083 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.487044    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.487300 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.487274    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhn8f\" (UniqueName: \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-kube-api-access-mhn8f\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.688100 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:45:03.688068    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11e81ed_de32_43bf_89d7_3503c50fe558.slice/crio-215a191e80b7d503ad577f4bdf7e1747ecaa4a1da0bd64fbf1e84f101bc32d04 WatchSource:0}: Error finding container 215a191e80b7d503ad577f4bdf7e1747ecaa4a1da0bd64fbf1e84f101bc32d04: Status 404 returned error can't find the container with id 215a191e80b7d503ad577f4bdf7e1747ecaa4a1da0bd64fbf1e84f101bc32d04
Oct 31 19:45:03.691282 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.691241    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:03.691399 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.691323    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:03.691399 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.691372    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:03.691509 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.691413    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"]
Oct 31 19:45:03.753704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.753678    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:03.853435 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.853342    5006 generic.go:356] "Generic (PLEG): container finished" podID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerID="020bb3510749aa54bc8092a10f701031135e63ce7e548ad9ca9b059028dc5d6b" exitCode=60
Oct 31 19:45:03.853435 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.853417    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" event={"ID":"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a","Type":"ContainerDied","Data":"020bb3510749aa54bc8092a10f701031135e63ce7e548ad9ca9b059028dc5d6b"}
Oct 31 19:45:03.853708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.853480    5006 scope.go:122] "RemoveContainer" containerID="26d511c5a66b06500e3510cf0fe5f66e9ceb98083528f8ca872858313e3e4f43"
Oct 31 19:45:03.853821 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.853803    5006 scope.go:122] "RemoveContainer" containerID="020bb3510749aa54bc8092a10f701031135e63ce7e548ad9ca9b059028dc5d6b"
Oct 31 19:45:03.874782 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:03.854138    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891265-dvjlc_odh-ai-gateway-infra(ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a"
Oct 31 19:45:03.874782 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.857344    5006 scope.go:122] "RemoveContainer" containerID="703b275eadbbb42f5bd78cfc1fac36f6c21bfaec29ace0ea553c52d730a0fcb9"
Oct 31 19:45:03.874782 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:03.857692    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95_odh-ai-gateway-infra(06c132c4-683a-4baa-9919-6aa73708457e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95" podUID="06c132c4-683a-4baa-9919-6aa73708457e"
Oct 31 19:45:03.874782 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.867840    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" event={"ID":"88e152e6-c6f7-4b7e-b262-8eb60682024d","Type":"ContainerStarted","Data":"e76f976d9caccfeffbe69c8c719abfceb3acd5129ceb43bdbbb6a9052e400dd6"}
Oct 31 19:45:03.874782 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.872083    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" event={"ID":"c630ed21-9b1b-44e6-b548-89eefc9a0acf","Type":"ContainerStarted","Data":"460fb1efd2321b81229c4f9ff8ba0de5a571565f60e8f2b0a6bc454d7db3c71b"}
Oct 31 19:45:03.877163 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.877134    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" event={"ID":"e11e81ed-de32-43bf-89d7-3503c50fe558","Type":"ContainerStarted","Data":"2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296"}
Oct 31 19:45:03.877305 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.877188    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" event={"ID":"e11e81ed-de32-43bf-89d7-3503c50fe558","Type":"ContainerStarted","Data":"215a191e80b7d503ad577f4bdf7e1747ecaa4a1da0bd64fbf1e84f101bc32d04"}
Oct 31 19:45:03.879449 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:03.879419    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" event={"ID":"69263e6c-ab4a-469e-ad65-101eb2bbe9c6","Type":"ContainerStarted","Data":"e4bd5233eb87c63c6f6af2a9d6f1181412993fae7295dbcda65f078e356b3e71"}
Oct 31 19:45:04.175589 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:45:04.175556    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447ea240_4362_41e5_b077_4daed43f4c41.slice/crio-6405dcd8db4f353af766f6bedb322309ede6d807c0686903536aad70bec679ef WatchSource:0}: Error finding container 6405dcd8db4f353af766f6bedb322309ede6d807c0686903536aad70bec679ef: Status 404 returned error can't find the container with id 6405dcd8db4f353af766f6bedb322309ede6d807c0686903536aad70bec679ef
Oct 31 19:45:04.178764 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.178723    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:04.178850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.178810    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:04.178850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.178839    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:04.180717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.180692    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"]
Oct 31 19:45:04.211894 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.211865    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:04.213405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.213378    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:45:04.213512 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.213436    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 19:45:04.308765 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.308703    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podStartSLOduration=2.308684167 podStartE2EDuration="2.308684167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:04.304212192 +0000 UTC m=+1722.833746246" watchObservedRunningTime="2026-10-31 19:45:04.308684167 +0000 UTC m=+1722.838218224"
Oct 31 19:45:04.917935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.916331    5006 generic.go:356] "Generic (PLEG): container finished" podID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerID="e4bd5233eb87c63c6f6af2a9d6f1181412993fae7295dbcda65f078e356b3e71" exitCode=60
Oct 31 19:45:04.917935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.916506    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" event={"ID":"69263e6c-ab4a-469e-ad65-101eb2bbe9c6","Type":"ContainerDied","Data":"e4bd5233eb87c63c6f6af2a9d6f1181412993fae7295dbcda65f078e356b3e71"}
Oct 31 19:45:04.917935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.916573    5006 scope.go:122] "RemoveContainer" containerID="5f8d39beaed5675df4b6a78bef96ddb960daa5768f08f32f1b30d3b90d8944cc"
Oct 31 19:45:04.917935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.917038    5006 scope.go:122] "RemoveContainer" containerID="e4bd5233eb87c63c6f6af2a9d6f1181412993fae7295dbcda65f078e356b3e71"
Oct 31 19:45:04.917935 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:04.917392    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg_odh-ai-gateway-infra(69263e6c-ab4a-469e-ad65-101eb2bbe9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6"
Oct 31 19:45:04.929350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.929321    5006 scope.go:122] "RemoveContainer" containerID="020bb3510749aa54bc8092a10f701031135e63ce7e548ad9ca9b059028dc5d6b"
Oct 31 19:45:04.929779 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:04.929748    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891265-dvjlc_odh-ai-gateway-infra(ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a"
Oct 31 19:45:04.930880 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.930849    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" event={"ID":"447ea240-4362-41e5-b077-4daed43f4c41","Type":"ContainerStarted","Data":"1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd"}
Oct 31 19:45:04.930990 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.930914    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" event={"ID":"447ea240-4362-41e5-b077-4daed43f4c41","Type":"ContainerStarted","Data":"6405dcd8db4f353af766f6bedb322309ede6d807c0686903536aad70bec679ef"}
Oct 31 19:45:04.933322 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.933297    5006 generic.go:356] "Generic (PLEG): container finished" podID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerID="e76f976d9caccfeffbe69c8c719abfceb3acd5129ceb43bdbbb6a9052e400dd6" exitCode=60
Oct 31 19:45:04.933436 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.933391    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" event={"ID":"88e152e6-c6f7-4b7e-b262-8eb60682024d","Type":"ContainerDied","Data":"e76f976d9caccfeffbe69c8c719abfceb3acd5129ceb43bdbbb6a9052e400dd6"}
Oct 31 19:45:04.933683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.933641    5006 scope.go:122] "RemoveContainer" containerID="e76f976d9caccfeffbe69c8c719abfceb3acd5129ceb43bdbbb6a9052e400dd6"
Oct 31 19:45:04.934001 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:04.933970    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t_odh-ai-gateway-infra(88e152e6-c6f7-4b7e-b262-8eb60682024d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d"
Oct 31 19:45:04.935683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.935644    5006 generic.go:356] "Generic (PLEG): container finished" podID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerID="460fb1efd2321b81229c4f9ff8ba0de5a571565f60e8f2b0a6bc454d7db3c71b" exitCode=60
Oct 31 19:45:04.935765 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.935746    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" event={"ID":"c630ed21-9b1b-44e6-b548-89eefc9a0acf","Type":"ContainerDied","Data":"460fb1efd2321b81229c4f9ff8ba0de5a571565f60e8f2b0a6bc454d7db3c71b"}
Oct 31 19:45:04.936031 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.936012    5006 scope.go:122] "RemoveContainer" containerID="460fb1efd2321b81229c4f9ff8ba0de5a571565f60e8f2b0a6bc454d7db3c71b"
Oct 31 19:45:04.936336 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:04.936315    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz_odh-ai-gateway-infra(c630ed21-9b1b-44e6-b548-89eefc9a0acf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf"
Oct 31 19:45:04.940994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.940970    5006 scope.go:122] "RemoveContainer" containerID="ed6deccc03898ad3be95854abbd0fb38e7d9e2f0be55b45ef4c2f88474f3304d"
Oct 31 19:45:04.955119 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:04.955093    5006 scope.go:122] "RemoveContainer" containerID="239e28a2bda400416b2565e8b035211bdd92cdc69b25f38b6c3e17e114ffc0e9"
Oct 31 19:45:05.214737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.213197    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:45:05.214737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.213278    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 19:45:05.433672 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:05.432889    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:45:05.519286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.519157    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" podStartSLOduration=2.519132031 podStartE2EDuration="2.519132031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:05.50314585 +0000 UTC m=+1724.032679905" watchObservedRunningTime="2026-10-31 19:45:05.519132031 +0000 UTC m=+1724.048666085"
Oct 31 19:45:05.662674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.660940    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:45:05.681767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.681737    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:45:05.757069 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.757036    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:05.758937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.758904    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:45:05.759101 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.758971    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 19:45:05.954672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.954044    5006 scope.go:122] "RemoveContainer" containerID="e4bd5233eb87c63c6f6af2a9d6f1181412993fae7295dbcda65f078e356b3e71"
Oct 31 19:45:05.954672 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:05.954414    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg_odh-ai-gateway-infra(69263e6c-ab4a-469e-ad65-101eb2bbe9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6"
Oct 31 19:45:05.958364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.957918    5006 scope.go:122] "RemoveContainer" containerID="e76f976d9caccfeffbe69c8c719abfceb3acd5129ceb43bdbbb6a9052e400dd6"
Oct 31 19:45:05.958364 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:05.958262    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t_odh-ai-gateway-infra(88e152e6-c6f7-4b7e-b262-8eb60682024d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d"
Oct 31 19:45:05.959594 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.959576    5006 scope.go:122] "RemoveContainer" containerID="460fb1efd2321b81229c4f9ff8ba0de5a571565f60e8f2b0a6bc454d7db3c71b"
Oct 31 19:45:05.959857 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:05.959838    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz_odh-ai-gateway-infra(c630ed21-9b1b-44e6-b548-89eefc9a0acf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf"
Oct 31 19:45:05.972215 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:05.972188    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:45:06.253999 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:06.253197    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:45:06.253999 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:06.253255    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 19:45:06.754441 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:06.754403    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:45:06.754744 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:06.754705    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 19:45:06.864678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:06.864633    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:07.230676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:07.217445    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:45:07.230676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:07.217511    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 19:45:07.754370 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:07.754322    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:45:07.754589 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:07.754399    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 19:45:08.213135 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:08.213091    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:45:08.213440 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:08.213410    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 19:45:08.755674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:08.754242    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:45:08.755674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:08.754319    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 19:45:09.212683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:09.212052    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:45:09.212683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:09.212130    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 19:45:09.754673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:09.754156    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:45:09.754673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:09.754235    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 19:45:10.214673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:10.214059    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:45:10.214673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:10.214124    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 19:45:10.754420 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:10.754376    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:45:10.754604 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:10.754459    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 19:45:11.619085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:11.619039    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:11.620723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:11.620696    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:11.624576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:11.624549    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:45:11.697367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:11.697337    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:12.110886 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:12.110850    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:12.111579 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:12.111552    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:12.112957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:12.112936    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:45:14.591192 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:14.591162    5006 scope.go:122] "RemoveContainer" containerID="e4bd5233eb87c63c6f6af2a9d6f1181412993fae7295dbcda65f078e356b3e71"
Oct 31 19:45:15.059376 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:15.059337    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" event={"ID":"69263e6c-ab4a-469e-ad65-101eb2bbe9c6","Type":"ContainerStarted","Data":"9c6c0c4ac8c2a3d1d9022047f9a14783b6ea8271b40e2e02df951af50ce8826a"}
Oct 31 19:45:15.599363 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:15.599333    5006 scope.go:122] "RemoveContainer" containerID="e76f976d9caccfeffbe69c8c719abfceb3acd5129ceb43bdbbb6a9052e400dd6"
Oct 31 19:45:15.599788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:15.599463    5006 scope.go:122] "RemoveContainer" containerID="703b275eadbbb42f5bd78cfc1fac36f6c21bfaec29ace0ea553c52d730a0fcb9"
Oct 31 19:45:16.068204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:16.068172    5006 generic.go:356] "Generic (PLEG): container finished" podID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerID="9c6c0c4ac8c2a3d1d9022047f9a14783b6ea8271b40e2e02df951af50ce8826a" exitCode=60
Oct 31 19:45:16.068370 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:16.068246    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" event={"ID":"69263e6c-ab4a-469e-ad65-101eb2bbe9c6","Type":"ContainerDied","Data":"9c6c0c4ac8c2a3d1d9022047f9a14783b6ea8271b40e2e02df951af50ce8826a"}
Oct 31 19:45:16.068370 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:16.068312    5006 scope.go:122] "RemoveContainer" containerID="e4bd5233eb87c63c6f6af2a9d6f1181412993fae7295dbcda65f078e356b3e71"
Oct 31 19:45:16.068738 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:16.068712    5006 scope.go:122] "RemoveContainer" containerID="9c6c0c4ac8c2a3d1d9022047f9a14783b6ea8271b40e2e02df951af50ce8826a"
Oct 31 19:45:16.069050 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:16.069024    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg_odh-ai-gateway-infra(69263e6c-ab4a-469e-ad65-101eb2bbe9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6"
Oct 31 19:45:16.070441 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:16.070415    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95" event={"ID":"06c132c4-683a-4baa-9919-6aa73708457e","Type":"ContainerStarted","Data":"f43d6ad926ffb0174316c6ed743c97b3313b51610d3c9f578a9fe75b6dd500b1"}
Oct 31 19:45:16.072484 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:16.072462    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" event={"ID":"88e152e6-c6f7-4b7e-b262-8eb60682024d","Type":"ContainerStarted","Data":"112cb0e7e3afc4f744c6e16d5daf793b8d0cd63060794bbfeee5557a375eb3f7"}
Oct 31 19:45:16.186890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:16.186804    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"]
Oct 31 19:45:16.313422 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:16.313364    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95" podStartSLOduration=16.313346993 podStartE2EDuration="16.313346993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:16.308762765 +0000 UTC m=+1734.838296816" watchObservedRunningTime="2026-10-31 19:45:16.313346993 +0000 UTC m=+1734.842881045"
Oct 31 19:45:16.560831 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:16.560775    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:17.083941 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.083903    5006 generic.go:356] "Generic (PLEG): container finished" podID="06c132c4-683a-4baa-9919-6aa73708457e" containerID="f43d6ad926ffb0174316c6ed743c97b3313b51610d3c9f578a9fe75b6dd500b1" exitCode=60
Oct 31 19:45:17.084344 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.083938    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95" event={"ID":"06c132c4-683a-4baa-9919-6aa73708457e","Type":"ContainerDied","Data":"f43d6ad926ffb0174316c6ed743c97b3313b51610d3c9f578a9fe75b6dd500b1"}
Oct 31 19:45:17.084344 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.084025    5006 scope.go:122] "RemoveContainer" containerID="703b275eadbbb42f5bd78cfc1fac36f6c21bfaec29ace0ea553c52d730a0fcb9"
Oct 31 19:45:17.084477 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.084450    5006 scope.go:122] "RemoveContainer" containerID="f43d6ad926ffb0174316c6ed743c97b3313b51610d3c9f578a9fe75b6dd500b1"
Oct 31 19:45:17.084830 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:17.084794    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95_odh-ai-gateway-infra(06c132c4-683a-4baa-9919-6aa73708457e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95" podUID="06c132c4-683a-4baa-9919-6aa73708457e"
Oct 31 19:45:17.086230 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.086212    5006 generic.go:356] "Generic (PLEG): container finished" podID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerID="112cb0e7e3afc4f744c6e16d5daf793b8d0cd63060794bbfeee5557a375eb3f7" exitCode=60
Oct 31 19:45:17.086304 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.086285    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" event={"ID":"88e152e6-c6f7-4b7e-b262-8eb60682024d","Type":"ContainerDied","Data":"112cb0e7e3afc4f744c6e16d5daf793b8d0cd63060794bbfeee5557a375eb3f7"}
Oct 31 19:45:17.086671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.086644    5006 scope.go:122] "RemoveContainer" containerID="112cb0e7e3afc4f744c6e16d5daf793b8d0cd63060794bbfeee5557a375eb3f7"
Oct 31 19:45:17.086956 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:17.086937    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t_odh-ai-gateway-infra(88e152e6-c6f7-4b7e-b262-8eb60682024d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d"
Oct 31 19:45:17.098149 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.098120    5006 scope.go:122] "RemoveContainer" containerID="e76f976d9caccfeffbe69c8c719abfceb3acd5129ceb43bdbbb6a9052e400dd6"
Oct 31 19:45:17.250584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.250559    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"
Oct 31 19:45:17.320704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.320632    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"]
Oct 31 19:45:17.381916 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.381887    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-kube-api-access-ft5gs\" (UniqueName: \"kubernetes.io/projected/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-kube-api-access-ft5gs\") pod \"69263e6c-ab4a-469e-ad65-101eb2bbe9c6\" (UID: \"69263e6c-ab4a-469e-ad65-101eb2bbe9c6\") "
Oct 31 19:45:17.382118 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.381961    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-maas-api-service-ca\") pod \"69263e6c-ab4a-469e-ad65-101eb2bbe9c6\" (UID: \"69263e6c-ab4a-469e-ad65-101eb2bbe9c6\") "
Oct 31 19:45:17.382396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.382362    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-maas-api-service-ca" pod "69263e6c-ab4a-469e-ad65-101eb2bbe9c6" (UID: "69263e6c-ab4a-469e-ad65-101eb2bbe9c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:17.382501 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.382472    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:17.384146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.384068    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-kube-api-access-ft5gs" pod "69263e6c-ab4a-469e-ad65-101eb2bbe9c6" (UID: "69263e6c-ab4a-469e-ad65-101eb2bbe9c6"). InnerVolumeSpecName "kube-api-access-ft5gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:17.439978 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.439943    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"]
Oct 31 19:45:17.483310 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:17.483275    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft5gs\" (UniqueName: \"kubernetes.io/projected/69263e6c-ab4a-469e-ad65-101eb2bbe9c6-kube-api-access-ft5gs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:18.099858 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.099825    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg" event={"ID":"69263e6c-ab4a-469e-ad65-101eb2bbe9c6","Type":"ContainerDied","Data":"b8d5519ee3f29ebd8419f9187d6e1bc5b32d7719e4755eb4977c2a0ae23a9fc8"}
Oct 31 19:45:18.100270 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.099869    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"
Oct 31 19:45:18.100270 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.099901    5006 scope.go:122] "RemoveContainer" containerID="9c6c0c4ac8c2a3d1d9022047f9a14783b6ea8271b40e2e02df951af50ce8826a"
Oct 31 19:45:18.324706 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.324679    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"
Oct 31 19:45:18.327102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.327082    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"
Oct 31 19:45:18.350996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.350906    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"]
Oct 31 19:45:18.355107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.355079    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891265-r8fbg"]
Oct 31 19:45:18.493620 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.493586    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c132c4-683a-4baa-9919-6aa73708457e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c132c4-683a-4baa-9919-6aa73708457e-maas-api-service-ca\") pod \"06c132c4-683a-4baa-9919-6aa73708457e\" (UID: \"06c132c4-683a-4baa-9919-6aa73708457e\") "
Oct 31 19:45:18.493827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.493637    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e152e6-c6f7-4b7e-b262-8eb60682024d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e152e6-c6f7-4b7e-b262-8eb60682024d-maas-api-service-ca\") pod \"88e152e6-c6f7-4b7e-b262-8eb60682024d\" (UID: \"88e152e6-c6f7-4b7e-b262-8eb60682024d\") "
Oct 31 19:45:18.493827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.493715    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e152e6-c6f7-4b7e-b262-8eb60682024d-kube-api-access-gslsl\" (UniqueName: \"kubernetes.io/projected/88e152e6-c6f7-4b7e-b262-8eb60682024d-kube-api-access-gslsl\") pod \"88e152e6-c6f7-4b7e-b262-8eb60682024d\" (UID: \"88e152e6-c6f7-4b7e-b262-8eb60682024d\") "
Oct 31 19:45:18.493827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.493786    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c132c4-683a-4baa-9919-6aa73708457e-kube-api-access-qrwnt\" (UniqueName: \"kubernetes.io/projected/06c132c4-683a-4baa-9919-6aa73708457e-kube-api-access-qrwnt\") pod \"06c132c4-683a-4baa-9919-6aa73708457e\" (UID: \"06c132c4-683a-4baa-9919-6aa73708457e\") "
Oct 31 19:45:18.494001 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.493958    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c132c4-683a-4baa-9919-6aa73708457e-maas-api-service-ca" pod "06c132c4-683a-4baa-9919-6aa73708457e" (UID: "06c132c4-683a-4baa-9919-6aa73708457e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:18.494120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.494097    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e152e6-c6f7-4b7e-b262-8eb60682024d-maas-api-service-ca" pod "88e152e6-c6f7-4b7e-b262-8eb60682024d" (UID: "88e152e6-c6f7-4b7e-b262-8eb60682024d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:18.494338 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.494320    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e152e6-c6f7-4b7e-b262-8eb60682024d-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:18.494396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.494343    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c132c4-683a-4baa-9919-6aa73708457e-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:18.496008 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.495973    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e152e6-c6f7-4b7e-b262-8eb60682024d-kube-api-access-gslsl" pod "88e152e6-c6f7-4b7e-b262-8eb60682024d" (UID: "88e152e6-c6f7-4b7e-b262-8eb60682024d"). InnerVolumeSpecName "kube-api-access-gslsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:18.496008 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.495987    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c132c4-683a-4baa-9919-6aa73708457e-kube-api-access-qrwnt" pod "06c132c4-683a-4baa-9919-6aa73708457e" (UID: "06c132c4-683a-4baa-9919-6aa73708457e"). InnerVolumeSpecName "kube-api-access-qrwnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:18.591017 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.590969    5006 scope.go:122] "RemoveContainer" containerID="020bb3510749aa54bc8092a10f701031135e63ce7e548ad9ca9b059028dc5d6b"
Oct 31 19:45:18.591723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.591694    5006 scope.go:122] "RemoveContainer" containerID="460fb1efd2321b81229c4f9ff8ba0de5a571565f60e8f2b0a6bc454d7db3c71b"
Oct 31 19:45:18.595580 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.595466    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gslsl\" (UniqueName: \"kubernetes.io/projected/88e152e6-c6f7-4b7e-b262-8eb60682024d-kube-api-access-gslsl\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:18.595580 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:18.595495    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrwnt\" (UniqueName: \"kubernetes.io/projected/06c132c4-683a-4baa-9919-6aa73708457e-kube-api-access-qrwnt\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:19.111569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.111530    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" event={"ID":"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a","Type":"ContainerStarted","Data":"c365c9b47bdbf46363c59527b5ebbf83ce7288dd0283aeb3c869aec42e32593d"}
Oct 31 19:45:19.112969 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.112950    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"
Oct 31 19:45:19.113095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.112999    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95" event={"ID":"06c132c4-683a-4baa-9919-6aa73708457e","Type":"ContainerDied","Data":"9038f81c979ba0eceebaef4178247eb93e90c29a3fe29f385bfd38a804ebea97"}
Oct 31 19:45:19.113095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.113061    5006 scope.go:122] "RemoveContainer" containerID="f43d6ad926ffb0174316c6ed743c97b3313b51610d3c9f578a9fe75b6dd500b1"
Oct 31 19:45:19.114453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.114424    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"
Oct 31 19:45:19.114453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.114439    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t" event={"ID":"88e152e6-c6f7-4b7e-b262-8eb60682024d","Type":"ContainerDied","Data":"dedee12d923421771b50a6624624656a42efbacee9667bdf9a365db322efaeee"}
Oct 31 19:45:19.116729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.116704    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" event={"ID":"c630ed21-9b1b-44e6-b548-89eefc9a0acf","Type":"ContainerStarted","Data":"596c999f02f5b4e9bbe1e0f30b864b1a06df37f18058058ffb621e1f07a41755"}
Oct 31 19:45:19.125177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.125155    5006 scope.go:122] "RemoveContainer" containerID="112cb0e7e3afc4f744c6e16d5daf793b8d0cd63060794bbfeee5557a375eb3f7"
Oct 31 19:45:19.342447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.342410    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"]
Oct 31 19:45:19.346953 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.346922    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891265-zrg2t"]
Oct 31 19:45:19.454875 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.454838    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"]
Oct 31 19:45:19.466830 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.466794    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891265-9dn95"]
Oct 31 19:45:19.599581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.599546    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c132c4-683a-4baa-9919-6aa73708457e" path="/var/lib/kubelet/pods/06c132c4-683a-4baa-9919-6aa73708457e/volumes"
Oct 31 19:45:19.600255 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.600221    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" path="/var/lib/kubelet/pods/69263e6c-ab4a-469e-ad65-101eb2bbe9c6/volumes"
Oct 31 19:45:19.600687 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:19.600670    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d" path="/var/lib/kubelet/pods/88e152e6-c6f7-4b7e-b262-8eb60682024d/volumes"
Oct 31 19:45:20.127170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:20.127136    5006 generic.go:356] "Generic (PLEG): container finished" podID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerID="596c999f02f5b4e9bbe1e0f30b864b1a06df37f18058058ffb621e1f07a41755" exitCode=60
Oct 31 19:45:20.127644 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:20.127214    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" event={"ID":"c630ed21-9b1b-44e6-b548-89eefc9a0acf","Type":"ContainerDied","Data":"596c999f02f5b4e9bbe1e0f30b864b1a06df37f18058058ffb621e1f07a41755"}
Oct 31 19:45:20.127644 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:20.127280    5006 scope.go:122] "RemoveContainer" containerID="460fb1efd2321b81229c4f9ff8ba0de5a571565f60e8f2b0a6bc454d7db3c71b"
Oct 31 19:45:20.127790 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:20.127732    5006 scope.go:122] "RemoveContainer" containerID="596c999f02f5b4e9bbe1e0f30b864b1a06df37f18058058ffb621e1f07a41755"
Oct 31 19:45:20.128114 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:20.128089    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz_odh-ai-gateway-infra(c630ed21-9b1b-44e6-b548-89eefc9a0acf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf"
Oct 31 19:45:20.136458 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:20.136405    5006 generic.go:356] "Generic (PLEG): container finished" podID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerID="c365c9b47bdbf46363c59527b5ebbf83ce7288dd0283aeb3c869aec42e32593d" exitCode=60
Oct 31 19:45:20.136628 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:20.136533    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" event={"ID":"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a","Type":"ContainerDied","Data":"c365c9b47bdbf46363c59527b5ebbf83ce7288dd0283aeb3c869aec42e32593d"}
Oct 31 19:45:20.136975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:20.136958    5006 scope.go:122] "RemoveContainer" containerID="c365c9b47bdbf46363c59527b5ebbf83ce7288dd0283aeb3c869aec42e32593d"
Oct 31 19:45:20.137303 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:20.137281    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891265-dvjlc_odh-ai-gateway-infra(ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a"
Oct 31 19:45:20.144135 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:20.144114    5006 scope.go:122] "RemoveContainer" containerID="020bb3510749aa54bc8092a10f701031135e63ce7e548ad9ca9b059028dc5d6b"
Oct 31 19:45:20.240403 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:20.240369    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"]
Oct 31 19:45:20.587621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:20.587584    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"]
Oct 31 19:45:21.256359 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.256282    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:21.351092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.351062    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"
Oct 31 19:45:21.354015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.353992    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"
Oct 31 19:45:21.476028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.475929    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:21.532639 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.532587    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-kube-api-access-6pf24\" (UniqueName: \"kubernetes.io/projected/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-kube-api-access-6pf24\") pod \"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a\" (UID: \"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a\") "
Oct 31 19:45:21.532868 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.532779    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c630ed21-9b1b-44e6-b548-89eefc9a0acf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c630ed21-9b1b-44e6-b548-89eefc9a0acf-maas-api-service-ca\") pod \"c630ed21-9b1b-44e6-b548-89eefc9a0acf\" (UID: \"c630ed21-9b1b-44e6-b548-89eefc9a0acf\") "
Oct 31 19:45:21.533044 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.533023    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-maas-api-service-ca\") pod \"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a\" (UID: \"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a\") "
Oct 31 19:45:21.533106 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.533067    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c630ed21-9b1b-44e6-b548-89eefc9a0acf-kube-api-access-xk4dw\" (UniqueName: \"kubernetes.io/projected/c630ed21-9b1b-44e6-b548-89eefc9a0acf-kube-api-access-xk4dw\") pod \"c630ed21-9b1b-44e6-b548-89eefc9a0acf\" (UID: \"c630ed21-9b1b-44e6-b548-89eefc9a0acf\") "
Oct 31 19:45:21.533199 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.533158    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c630ed21-9b1b-44e6-b548-89eefc9a0acf-maas-api-service-ca" pod "c630ed21-9b1b-44e6-b548-89eefc9a0acf" (UID: "c630ed21-9b1b-44e6-b548-89eefc9a0acf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:21.533481 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.533429    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-maas-api-service-ca" pod "ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" (UID: "ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:21.533723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.533701    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.533817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.533729    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c630ed21-9b1b-44e6-b548-89eefc9a0acf-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.535108 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.535083    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-kube-api-access-6pf24" pod "ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" (UID: "ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a"). InnerVolumeSpecName "kube-api-access-6pf24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:21.535591 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.535570    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c630ed21-9b1b-44e6-b548-89eefc9a0acf-kube-api-access-xk4dw" pod "c630ed21-9b1b-44e6-b548-89eefc9a0acf" (UID: "c630ed21-9b1b-44e6-b548-89eefc9a0acf"). InnerVolumeSpecName "kube-api-access-xk4dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:21.635032 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.635003    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk4dw\" (UniqueName: \"kubernetes.io/projected/c630ed21-9b1b-44e6-b548-89eefc9a0acf-kube-api-access-xk4dw\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.635032 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:21.635033    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pf24\" (UniqueName: \"kubernetes.io/projected/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a-kube-api-access-6pf24\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:22.157851 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:22.157757    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"
Oct 31 19:45:22.157851 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:22.157790    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz" event={"ID":"c630ed21-9b1b-44e6-b548-89eefc9a0acf","Type":"ContainerDied","Data":"d785a1aa85014ce6171ab502d563b8183ed98c1e08b7c0464d593af4b436e1ca"}
Oct 31 19:45:22.158076 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:22.157864    5006 scope.go:122] "RemoveContainer" containerID="596c999f02f5b4e9bbe1e0f30b864b1a06df37f18058058ffb621e1f07a41755"
Oct 31 19:45:22.159437 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:22.159404    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"
Oct 31 19:45:22.159539 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:22.159427    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc" event={"ID":"ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a","Type":"ContainerDied","Data":"b15f7ce82096d03c0182bd25b5f6a443b84ddc574b690dbea5cfc67aee73b2ff"}
Oct 31 19:45:22.168168 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:22.168146    5006 scope.go:122] "RemoveContainer" containerID="c365c9b47bdbf46363c59527b5ebbf83ce7288dd0283aeb3c869aec42e32593d"
Oct 31 19:45:22.324914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:22.324878    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"]
Oct 31 19:45:22.374633 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:22.374598    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-dvjlc"]
Oct 31 19:45:22.484414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:22.484329    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"]
Oct 31 19:45:22.489226 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:22.489174    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891265-5r2pz"]
Oct 31 19:45:23.598372 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:23.598340    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" path="/var/lib/kubelet/pods/c630ed21-9b1b-44e6-b548-89eefc9a0acf/volumes"
Oct 31 19:45:23.598842 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:23.598826    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" path="/var/lib/kubelet/pods/ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a/volumes"
Oct 31 19:45:24.243696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:24.243643    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:45:26.300439 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:26.300388    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:30.379869 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:30.379831    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:31.129799 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:31.129769    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:35.394275 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:45:35.394225    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:45:36.072298 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:36.072259    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:37.575435 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.575346    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"]
Oct 31 19:45:37.575956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.575941    5006 cpu_manager.go:414] "containerMap: removing container" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerName="cleanup"
Oct 31 19:45:37.576005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.575957    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerName="cleanup"
Oct 31 19:45:37.576005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.575972    5006 cpu_manager.go:414] "containerMap: removing container" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerName="cleanup"
Oct 31 19:45:37.576005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.575978    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerName="cleanup"
Oct 31 19:45:37.576005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.575986    5006 cpu_manager.go:414] "containerMap: removing container" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerName="cleanup"
Oct 31 19:45:37.576005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.575992    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerName="cleanup"
Oct 31 19:45:37.576005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576007    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576012    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576023    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576028    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576037    5006 cpu_manager.go:414] "containerMap: removing container" podUID="06c132c4-683a-4baa-9919-6aa73708457e" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576042    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c132c4-683a-4baa-9919-6aa73708457e" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576053    5006 cpu_manager.go:414] "containerMap: removing container" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576057    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576067    5006 cpu_manager.go:414] "containerMap: removing container" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576071    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576078    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576082    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576093    5006 cpu_manager.go:414] "containerMap: removing container" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576098    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576109    5006 cpu_manager.go:414] "containerMap: removing container" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576113    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576138    5006 cpu_manager.go:414] "containerMap: removing container" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576143    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576153    5006 cpu_manager.go:414] "containerMap: removing container" podUID="06c132c4-683a-4baa-9919-6aa73708457e" containerName="cleanup"
Oct 31 19:45:37.576180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576158    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c132c4-683a-4baa-9919-6aa73708457e" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576242    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576253    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576259    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2e5f31-e6c9-48f0-a82f-03fb38e8f26a" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576266    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576276    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576285    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c132c4-683a-4baa-9919-6aa73708457e" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576293    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576302    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576308    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c630ed21-9b1b-44e6-b548-89eefc9a0acf" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576317    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576325    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="69263e6c-ab4a-469e-ad65-101eb2bbe9c6" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576334    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576343    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c132c4-683a-4baa-9919-6aa73708457e" containerName="cleanup"
Oct 31 19:45:37.576870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.576354    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c132c4-683a-4baa-9919-6aa73708457e" containerName="cleanup"
Oct 31 19:45:37.580085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.580067    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:37.633417 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.633383    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"]
Oct 31 19:45:37.679573 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.679535    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:45:37.679573 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.679564    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:45:37.679830 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.679570    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:45:37.683015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.682987    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:45:37.720514 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.720466    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkwb5\" (UniqueName: \"kubernetes.io/projected/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-kube-api-access-nkwb5\") pod \"maas-api-revoke-sub-9993bb24-t2d9c\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:37.720748 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.720607    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9993bb24-t2d9c\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:37.720829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.720751    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9993bb24-t2d9c\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:37.821980 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.821934    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9993bb24-t2d9c\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:37.822157 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.822019    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkwb5\" (UniqueName: \"kubernetes.io/projected/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-kube-api-access-nkwb5\") pod \"maas-api-revoke-sub-9993bb24-t2d9c\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:37.822157 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.822134    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9993bb24-t2d9c\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:37.822792 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.822768    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9993bb24-t2d9c\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:37.824400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.824371    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9993bb24-t2d9c\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:37.879779 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.879742    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkwb5\" (UniqueName: \"kubernetes.io/projected/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-kube-api-access-nkwb5\") pod \"maas-api-revoke-sub-9993bb24-t2d9c\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:37.895098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:37.895064    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:38.268816 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:45:38.268771    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c44c19_46af_4b3a_b2c0_57cda3fb8b60.slice/crio-90677728d1f2b2abeb3e87bc278242da3a4cbb7dc1eaaaa31bade1c6889c69dd WatchSource:0}: Error finding container 90677728d1f2b2abeb3e87bc278242da3a4cbb7dc1eaaaa31bade1c6889c69dd: Status 404 returned error can't find the container with id 90677728d1f2b2abeb3e87bc278242da3a4cbb7dc1eaaaa31bade1c6889c69dd
Oct 31 19:45:38.272533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:38.272508    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"]
Oct 31 19:45:38.298280 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:38.298243    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c" event={"ID":"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60","Type":"ContainerStarted","Data":"90677728d1f2b2abeb3e87bc278242da3a4cbb7dc1eaaaa31bade1c6889c69dd"}
Oct 31 19:45:39.310256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:39.310005    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c" event={"ID":"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60","Type":"ContainerStarted","Data":"da9f0b31650c2a1387871acd91c7ee873dec3682c2142e23944d9fab0af7fc1e"}
Oct 31 19:45:40.319075 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.319040    5006 generic.go:356] "Generic (PLEG): container finished" podID="a8c44c19-46af-4b3a-b2c0-57cda3fb8b60" containerID="da9f0b31650c2a1387871acd91c7ee873dec3682c2142e23944d9fab0af7fc1e" exitCode=0
Oct 31 19:45:40.319488 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.319133    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c" event={"ID":"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60","Type":"ContainerDied","Data":"da9f0b31650c2a1387871acd91c7ee873dec3682c2142e23944d9fab0af7fc1e"}
Oct 31 19:45:40.469789 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.469761    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:40.555076 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.555032    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-kube-api-access-nkwb5\" (UniqueName: \"kubernetes.io/projected/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-kube-api-access-nkwb5\") pod \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") "
Oct 31 19:45:40.555236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.555184    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-xks-ca\") pod \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") "
Oct 31 19:45:40.555299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.555287    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-openshift-ca\") pod \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\" (UID: \"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60\") "
Oct 31 19:45:40.555677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.555628    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-openshift-ca" pod "a8c44c19-46af-4b3a-b2c0-57cda3fb8b60" (UID: "a8c44c19-46af-4b3a-b2c0-57cda3fb8b60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:40.555827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.555811    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:40.557265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.557239    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-xks-ca" pod "a8c44c19-46af-4b3a-b2c0-57cda3fb8b60" (UID: "a8c44c19-46af-4b3a-b2c0-57cda3fb8b60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:45:40.557358 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.557277    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-kube-api-access-nkwb5" pod "a8c44c19-46af-4b3a-b2c0-57cda3fb8b60" (UID: "a8c44c19-46af-4b3a-b2c0-57cda3fb8b60"). InnerVolumeSpecName "kube-api-access-nkwb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:40.657458 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.657419    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkwb5\" (UniqueName: \"kubernetes.io/projected/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-kube-api-access-nkwb5\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:40.657458 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:40.657455    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:41.053949 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:41.053869    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:41.328306 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:41.328218    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"
Oct 31 19:45:41.328306 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:41.328244    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c" event={"ID":"a8c44c19-46af-4b3a-b2c0-57cda3fb8b60","Type":"ContainerDied","Data":"90677728d1f2b2abeb3e87bc278242da3a4cbb7dc1eaaaa31bade1c6889c69dd"}
Oct 31 19:45:41.328306 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:41.328294    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90677728d1f2b2abeb3e87bc278242da3a4cbb7dc1eaaaa31bade1c6889c69dd"
Oct 31 19:45:45.989565 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:45.989521    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:50.857346 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:50.857302    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:52.392355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.392314    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"]
Oct 31 19:45:52.393149 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.393130    5006 cpu_manager.go:414] "containerMap: removing container" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerName="cleanup"
Oct 31 19:45:52.393201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.393151    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerName="cleanup"
Oct 31 19:45:52.393201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.393182    5006 cpu_manager.go:414] "containerMap: removing container" podUID="06c132c4-683a-4baa-9919-6aa73708457e" containerName="cleanup"
Oct 31 19:45:52.393201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.393187    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c132c4-683a-4baa-9919-6aa73708457e" containerName="cleanup"
Oct 31 19:45:52.393201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.393196    5006 cpu_manager.go:414] "containerMap: removing container" podUID="a8c44c19-46af-4b3a-b2c0-57cda3fb8b60" containerName="revoke-keys"
Oct 31 19:45:52.393201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.393202    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c44c19-46af-4b3a-b2c0-57cda3fb8b60" containerName="revoke-keys"
Oct 31 19:45:52.393353 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.393303    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e152e6-c6f7-4b7e-b262-8eb60682024d" containerName="cleanup"
Oct 31 19:45:52.393353 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.393321    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c44c19-46af-4b3a-b2c0-57cda3fb8b60" containerName="revoke-keys"
Oct 31 19:45:52.397193 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.397169    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:52.450536 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.450500    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"]
Oct 31 19:45:52.495456 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.495416    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:45:52.495669 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.495461    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:45:52.495669 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.495500    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:45:52.500162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.500138    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:45:52.501936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.501911    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651f5bab-ngp4x\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:52.502052 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.501952    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651f5bab-ngp4x\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:52.502099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.502075    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p65s\" (UniqueName: \"kubernetes.io/projected/0784a054-7784-4de5-8366-20c28db11ca2-kube-api-access-2p65s\") pod \"maas-api-revoke-sub-651f5bab-ngp4x\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:52.603040 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.602997    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651f5bab-ngp4x\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:52.603259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.603100    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p65s\" (UniqueName: \"kubernetes.io/projected/0784a054-7784-4de5-8366-20c28db11ca2-kube-api-access-2p65s\") pod \"maas-api-revoke-sub-651f5bab-ngp4x\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:52.603327 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.603287    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651f5bab-ngp4x\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:52.603929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.603903    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651f5bab-ngp4x\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:52.605581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.605560    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651f5bab-ngp4x\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:52.660526 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.660454    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p65s\" (UniqueName: \"kubernetes.io/projected/0784a054-7784-4de5-8366-20c28db11ca2-kube-api-access-2p65s\") pod \"maas-api-revoke-sub-651f5bab-ngp4x\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:52.712968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:52.712931    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:53.085357 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:45:53.085316    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0784a054_7784_4de5_8366_20c28db11ca2.slice/crio-7b2ac4d67f3ca4cca72af8794a36c5f00ed7480411908bbdb94246add88f240c WatchSource:0}: Error finding container 7b2ac4d67f3ca4cca72af8794a36c5f00ed7480411908bbdb94246add88f240c: Status 404 returned error can't find the container with id 7b2ac4d67f3ca4cca72af8794a36c5f00ed7480411908bbdb94246add88f240c
Oct 31 19:45:53.089681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:53.089619    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"]
Oct 31 19:45:53.417593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:53.417557    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x" event={"ID":"0784a054-7784-4de5-8366-20c28db11ca2","Type":"ContainerStarted","Data":"7b2ac4d67f3ca4cca72af8794a36c5f00ed7480411908bbdb94246add88f240c"}
Oct 31 19:45:54.427536 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:54.427501    5006 generic.go:356] "Generic (PLEG): container finished" podID="0784a054-7784-4de5-8366-20c28db11ca2" containerID="4a304b94db2c76c3bdee2bfc32ff51271a5e61a15f7d9c1ffbf8c47910255d57" exitCode=0
Oct 31 19:45:54.427960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:54.427583    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x" event={"ID":"0784a054-7784-4de5-8366-20c28db11ca2","Type":"ContainerDied","Data":"4a304b94db2c76c3bdee2bfc32ff51271a5e61a15f7d9c1ffbf8c47910255d57"}
Oct 31 19:45:55.608861 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.608824    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:55.637167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.637130    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-xks-ca\") pod \"0784a054-7784-4de5-8366-20c28db11ca2\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") "
Oct 31 19:45:55.637374 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.637238    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0784a054-7784-4de5-8366-20c28db11ca2-kube-api-access-2p65s\" (UniqueName: \"kubernetes.io/projected/0784a054-7784-4de5-8366-20c28db11ca2-kube-api-access-2p65s\") pod \"0784a054-7784-4de5-8366-20c28db11ca2\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") "
Oct 31 19:45:55.637374 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.637288    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-openshift-ca\") pod \"0784a054-7784-4de5-8366-20c28db11ca2\" (UID: \"0784a054-7784-4de5-8366-20c28db11ca2\") "
Oct 31 19:45:55.637931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.637855    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-openshift-ca" pod "0784a054-7784-4de5-8366-20c28db11ca2" (UID: "0784a054-7784-4de5-8366-20c28db11ca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:55.639696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.639623    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-xks-ca" pod "0784a054-7784-4de5-8366-20c28db11ca2" (UID: "0784a054-7784-4de5-8366-20c28db11ca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:45:55.643785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.643744    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0784a054-7784-4de5-8366-20c28db11ca2-kube-api-access-2p65s" pod "0784a054-7784-4de5-8366-20c28db11ca2" (UID: "0784a054-7784-4de5-8366-20c28db11ca2"). InnerVolumeSpecName "kube-api-access-2p65s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:55.675328 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.675275    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:55.739203 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.739102    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.739203 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.739141    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p65s\" (UniqueName: \"kubernetes.io/projected/0784a054-7784-4de5-8366-20c28db11ca2-kube-api-access-2p65s\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.739203 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:55.739155    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0784a054-7784-4de5-8366-20c28db11ca2-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:56.448194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:56.448163    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"
Oct 31 19:45:56.448194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:56.448182    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x" event={"ID":"0784a054-7784-4de5-8366-20c28db11ca2","Type":"ContainerDied","Data":"7b2ac4d67f3ca4cca72af8794a36c5f00ed7480411908bbdb94246add88f240c"}
Oct 31 19:45:56.448439 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:45:56.448237    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b2ac4d67f3ca4cca72af8794a36c5f00ed7480411908bbdb94246add88f240c"
Oct 31 19:46:00.470204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:00.470155    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:04.201046 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.201013    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"]
Oct 31 19:46:04.201575 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.201559    5006 cpu_manager.go:414] "containerMap: removing container" podUID="0784a054-7784-4de5-8366-20c28db11ca2" containerName="revoke-keys"
Oct 31 19:46:04.201620 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.201575    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0784a054-7784-4de5-8366-20c28db11ca2" containerName="revoke-keys"
Oct 31 19:46:04.201743 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.201733    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0784a054-7784-4de5-8366-20c28db11ca2" containerName="revoke-keys"
Oct 31 19:46:04.205738 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.205718    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.216541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.216511    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:46:04.336690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.336632    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.336690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.336692    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.336914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.336783    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.336914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.336860    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.336914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.336890    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d6fc0d-24c8-4770-8a37-101c0775b924-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.337015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.336925    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.337049 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.337019    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.337082 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.337045    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.337209 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.337195    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmlx7\" (UniqueName: \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-kube-api-access-hmlx7\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.438740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.438702    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.438740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.438742    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.439012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.438767    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.439012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.438815    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.439012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.438842    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d6fc0d-24c8-4770-8a37-101c0775b924-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.439012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.438879    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.439012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.438937    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.439910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.439041    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.439910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.439187    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.439910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.439396    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.439910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.439432    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmlx7\" (UniqueName: \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-kube-api-access-hmlx7\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.439910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.439806    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.440781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.439939    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d6fc0d-24c8-4770-8a37-101c0775b924-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.440781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.439951    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.441598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.441574    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.442185 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.442162    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.508640 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.508557    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"]
Oct 31 19:46:04.557710 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.554239    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7cfafd-openshift-default-dockercfg-96gwp\""
Oct 31 19:46:04.557710 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.557442    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.557956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.557743    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmlx7\" (UniqueName: \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-kube-api-access-hmlx7\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:04.820311 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:04.820213    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:05.270289 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:46:05.270257    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d6fc0d_24c8_4770_8a37_101c0775b924.slice/crio-d66e5dd92727228114ebabae4f163e136a599562e68c0077661b913660c04d3e WatchSource:0}: Error finding container d66e5dd92727228114ebabae4f163e136a599562e68c0077661b913660c04d3e: Status 404 returned error can't find the container with id d66e5dd92727228114ebabae4f163e136a599562e68c0077661b913660c04d3e
Oct 31 19:46:05.273218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:05.273170    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:46:05.273350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:05.273246    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:46:05.273350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:05.273276    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:46:05.275198 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:05.275158    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"]
Oct 31 19:46:05.398712 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:46:05.398628    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:46:05.516153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:05.516086    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" event={"ID":"23d6fc0d-24c8-4770-8a37-101c0775b924","Type":"ContainerStarted","Data":"0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6"}
Oct 31 19:46:05.516366 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:05.516186    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" event={"ID":"23d6fc0d-24c8-4770-8a37-101c0775b924","Type":"ContainerStarted","Data":"d66e5dd92727228114ebabae4f163e136a599562e68c0077661b913660c04d3e"}
Oct 31 19:46:05.585816 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:05.585702    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:05.635671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:05.635596    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" podStartSLOduration=1.6355764769999999 podStartE2EDuration="1.635576477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:05.633861908 +0000 UTC m=+1784.163395961" watchObservedRunningTime="2026-10-31 19:46:05.635576477 +0000 UTC m=+1784.165110530"
Oct 31 19:46:05.820874 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:05.820762    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:06.822638 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:06.822603    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 19:46:06.823107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:06.822699    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 19:46:07.821167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:07.821131    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 19:46:07.821359 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:07.821193    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 19:46:08.821133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:08.821092    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 19:46:08.821559 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:08.821149    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 19:46:09.821183 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:09.821127    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 19:46:09.821772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:09.821192    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 19:46:10.483769 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:10.483715    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:10.820791 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:10.820686    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 19:46:10.820791 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:10.820749    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 19:46:11.821388 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:11.821345    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 19:46:11.821833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:11.821429    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 19:46:12.826014 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:12.825978    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:12.826503 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:12.826485    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:12.827353 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:12.827328    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:46:14.669571 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:14.669518    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:15.416928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:15.416821    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:17.457250 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.457212    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"]
Oct 31 19:46:17.461094 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.461078    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.568943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.568902    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"]
Oct 31 19:46:17.628681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.628609    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.628896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.628766    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82dtc\" (UniqueName: \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-kube-api-access-82dtc\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.628896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.628871    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.629004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.628904    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.629050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.629020    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.629050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.629041    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b99532-ceaf-4202-b988-c4a88e738313-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.629147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.629079    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.629201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.629180    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.629254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.629224    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b99532-ceaf-4202-b988-c4a88e738313-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730087    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730125    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b99532-ceaf-4202-b988-c4a88e738313-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730390 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730186    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730390 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730222    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730496 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730392    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b99532-ceaf-4202-b988-c4a88e738313-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730551 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730541    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730608 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730543    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730608 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730597    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730744 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730618    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82dtc\" (UniqueName: \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-kube-api-access-82dtc\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730807 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730740    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730807 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730775    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.730913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.730883    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.731180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.731147    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.731273 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.731195    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b99532-ceaf-4202-b988-c4a88e738313-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.732908 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.732882    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.733060 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.733043    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b99532-ceaf-4202-b988-c4a88e738313-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.788589 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.788556    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:17.788776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:17.788644    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82dtc\" (UniqueName: \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-kube-api-access-82dtc\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:18.076154 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:18.076058    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:18.501840 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:46:18.501809    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b99532_ceaf_4202_b988_c4a88e738313.slice/crio-62284eb902ad030e6abdc02b5ced709ef1b80315700e83b99367b6da56916683 WatchSource:0}: Error finding container 62284eb902ad030e6abdc02b5ced709ef1b80315700e83b99367b6da56916683: Status 404 returned error can't find the container with id 62284eb902ad030e6abdc02b5ced709ef1b80315700e83b99367b6da56916683
Oct 31 19:46:18.504571 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:18.504533    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:46:18.504702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:18.504608    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:46:18.504702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:18.504638    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:46:18.505717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:18.505693    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"]
Oct 31 19:46:18.625335 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:18.625291    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" event={"ID":"85b99532-ceaf-4202-b988-c4a88e738313","Type":"ContainerStarted","Data":"c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8"}
Oct 31 19:46:18.625538 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:18.625364    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" event={"ID":"85b99532-ceaf-4202-b988-c4a88e738313","Type":"ContainerStarted","Data":"62284eb902ad030e6abdc02b5ced709ef1b80315700e83b99367b6da56916683"}
Oct 31 19:46:18.746002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:18.745949    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" podStartSLOduration=1.745936708 podStartE2EDuration="1.745936708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:18.743401587 +0000 UTC m=+1797.272935660" watchObservedRunningTime="2026-10-31 19:46:18.745936708 +0000 UTC m=+1797.275470760"
Oct 31 19:46:18.997323 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:18.997278    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"]
Oct 31 19:46:19.001574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.001554    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.054936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.054864    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:46:19.055287 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.055266    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:46:19.055429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.055417    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:46:19.055631 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.055616    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:46:19.057201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.057166    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5389803-9527-4b79-acbe-da167c749990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc999293-nqdhj\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.057393 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.057373    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5389803-9527-4b79-acbe-da167c749990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc999293-nqdhj\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.057523 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.057507    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkcpf\" (UniqueName: \"kubernetes.io/projected/e5389803-9527-4b79-acbe-da167c749990-kube-api-access-lkcpf\") pod \"maas-api-revoke-sub-bc999293-nqdhj\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.063691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.063627    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"]
Oct 31 19:46:19.076867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.076828    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:19.078406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.078355    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 19:46:19.078551 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.078471    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 19:46:19.158542 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.158508    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5389803-9527-4b79-acbe-da167c749990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc999293-nqdhj\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.158779 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.158554    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5389803-9527-4b79-acbe-da167c749990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc999293-nqdhj\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.158779 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.158592    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkcpf\" (UniqueName: \"kubernetes.io/projected/e5389803-9527-4b79-acbe-da167c749990-kube-api-access-lkcpf\") pod \"maas-api-revoke-sub-bc999293-nqdhj\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.159227 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.159205    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5389803-9527-4b79-acbe-da167c749990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc999293-nqdhj\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.161031 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.161011    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5389803-9527-4b79-acbe-da167c749990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc999293-nqdhj\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.222142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.222105    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkcpf\" (UniqueName: \"kubernetes.io/projected/e5389803-9527-4b79-acbe-da167c749990-kube-api-access-lkcpf\") pod \"maas-api-revoke-sub-bc999293-nqdhj\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.321002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.320896    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:19.697718 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:46:19.695936    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5389803_9527_4b79_acbe_da167c749990.slice/crio-57af0e9ce81b0432035c48692118beb31aa1717ca90ba0c762e8722b1a5b27b0 WatchSource:0}: Error finding container 57af0e9ce81b0432035c48692118beb31aa1717ca90ba0c762e8722b1a5b27b0: Status 404 returned error can't find the container with id 57af0e9ce81b0432035c48692118beb31aa1717ca90ba0c762e8722b1a5b27b0
Oct 31 19:46:19.708690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:19.705398    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"]
Oct 31 19:46:20.076840 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:20.076748    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 19:46:20.076840 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:20.076808    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 19:46:20.442299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:20.442263    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:20.647469 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:20.647424    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj" event={"ID":"e5389803-9527-4b79-acbe-da167c749990","Type":"ContainerStarted","Data":"ae4bc783e5d9eac988640a2b2dbc091c1d7dd5bc92acee32af6e76f07e5787bb"}
Oct 31 19:46:20.647662 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:20.647488    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj" event={"ID":"e5389803-9527-4b79-acbe-da167c749990","Type":"ContainerStarted","Data":"57af0e9ce81b0432035c48692118beb31aa1717ca90ba0c762e8722b1a5b27b0"}
Oct 31 19:46:20.770295 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:20.770190    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj" podStartSLOduration=2.770174791 podStartE2EDuration="2.770174791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:20.766591747 +0000 UTC m=+1799.296125791" watchObservedRunningTime="2026-10-31 19:46:20.770174791 +0000 UTC m=+1799.299708861"
Oct 31 19:46:21.076638 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:21.076553    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 19:46:21.076638 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:21.076616    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 19:46:21.589971 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:21.589928    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:21.656280 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:21.656247    5006 generic.go:356] "Generic (PLEG): container finished" podID="e5389803-9527-4b79-acbe-da167c749990" containerID="ae4bc783e5d9eac988640a2b2dbc091c1d7dd5bc92acee32af6e76f07e5787bb" exitCode=0
Oct 31 19:46:21.656449 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:21.656323    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj" event={"ID":"e5389803-9527-4b79-acbe-da167c749990","Type":"ContainerDied","Data":"ae4bc783e5d9eac988640a2b2dbc091c1d7dd5bc92acee32af6e76f07e5787bb"}
Oct 31 19:46:21.939732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:21.939699    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:46:21.948393 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:21.948355    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:46:22.076560 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:22.076519    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 19:46:22.076815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:22.076589    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 19:46:22.864315 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:22.864287    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:22.927459 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:22.927366    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5389803-9527-4b79-acbe-da167c749990-kube-api-access-lkcpf\" (UniqueName: \"kubernetes.io/projected/e5389803-9527-4b79-acbe-da167c749990-kube-api-access-lkcpf\") pod \"e5389803-9527-4b79-acbe-da167c749990\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") "
Oct 31 19:46:22.927459 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:22.927429    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5389803-9527-4b79-acbe-da167c749990-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5389803-9527-4b79-acbe-da167c749990-maas-api-xks-ca\") pod \"e5389803-9527-4b79-acbe-da167c749990\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") "
Oct 31 19:46:22.927727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:22.927524    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5389803-9527-4b79-acbe-da167c749990-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5389803-9527-4b79-acbe-da167c749990-maas-api-openshift-ca\") pod \"e5389803-9527-4b79-acbe-da167c749990\" (UID: \"e5389803-9527-4b79-acbe-da167c749990\") "
Oct 31 19:46:22.927870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:22.927835    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5389803-9527-4b79-acbe-da167c749990-maas-api-openshift-ca" pod "e5389803-9527-4b79-acbe-da167c749990" (UID: "e5389803-9527-4b79-acbe-da167c749990"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:46:22.928071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:22.928056    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5389803-9527-4b79-acbe-da167c749990-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:22.929694 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:22.929670    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5389803-9527-4b79-acbe-da167c749990-kube-api-access-lkcpf" pod "e5389803-9527-4b79-acbe-da167c749990" (UID: "e5389803-9527-4b79-acbe-da167c749990"). InnerVolumeSpecName "kube-api-access-lkcpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:22.929758 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:22.929718    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5389803-9527-4b79-acbe-da167c749990-maas-api-xks-ca" pod "e5389803-9527-4b79-acbe-da167c749990" (UID: "e5389803-9527-4b79-acbe-da167c749990"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:46:23.029228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:23.029183    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkcpf\" (UniqueName: \"kubernetes.io/projected/e5389803-9527-4b79-acbe-da167c749990-kube-api-access-lkcpf\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:23.029228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:23.029221    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5389803-9527-4b79-acbe-da167c749990-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:23.077368 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:23.077328    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 19:46:23.077554 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:23.077396    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 19:46:23.673121 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:23.673092    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"
Oct 31 19:46:23.673261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:23.673120    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj" event={"ID":"e5389803-9527-4b79-acbe-da167c749990","Type":"ContainerDied","Data":"57af0e9ce81b0432035c48692118beb31aa1717ca90ba0c762e8722b1a5b27b0"}
Oct 31 19:46:23.673261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:23.673169    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57af0e9ce81b0432035c48692118beb31aa1717ca90ba0c762e8722b1a5b27b0"
Oct 31 19:46:24.076907 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:24.076812    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 19:46:24.076907 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:24.076874    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 19:46:25.076827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:25.076789    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 19:46:25.077037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:25.076855    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 19:46:25.392113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:25.392076    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:27.063116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:27.063029    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:27.063624 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:27.063399    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:27.066129 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:27.066108    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 19:46:28.099565 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.099517    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"]
Oct 31 19:46:28.100176 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.100160    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e5389803-9527-4b79-acbe-da167c749990" containerName="revoke-keys"
Oct 31 19:46:28.100217 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.100177    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5389803-9527-4b79-acbe-da167c749990" containerName="revoke-keys"
Oct 31 19:46:28.100268 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.100259    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5389803-9527-4b79-acbe-da167c749990" containerName="revoke-keys"
Oct 31 19:46:28.105093 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.105068    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.158866 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.158826    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"]
Oct 31 19:46:28.194830 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.194792    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-tmp-dir\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.195008 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.194929    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7z7\" (UniqueName: \"kubernetes.io/projected/e9f8518c-5924-44d3-8a07-e6578830d3b6-kube-api-access-7z7z7\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.195008 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.194984    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-home\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.195086 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.195067    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-dshm\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.195131 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.195102    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8518c-5924-44d3-8a07-e6578830d3b6-tls-certs\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.195131 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.195119    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-model-cache\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.203470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.203442    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"distinct-2-w3-662be1-kserve-self-signed-certs\""
Oct 31 19:46:28.279899 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.279855    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:28.296868 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.296827    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7z7\" (UniqueName: \"kubernetes.io/projected/e9f8518c-5924-44d3-8a07-e6578830d3b6-kube-api-access-7z7z7\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.297088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.296888    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-home\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.297088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.296933    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-dshm\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.297088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.296965    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8518c-5924-44d3-8a07-e6578830d3b6-tls-certs\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.297088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.296988    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-model-cache\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.297381 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.297097    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-tmp-dir\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.297592 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.297556    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-home\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.297722 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.297600    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-tmp-dir\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.297722 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.297617    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-model-cache\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.299863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.299839    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8518c-5924-44d3-8a07-e6578830d3b6-tls-certs\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.301943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.301912    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-dshm\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.359729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.359623    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7z7\" (UniqueName: \"kubernetes.io/projected/e9f8518c-5924-44d3-8a07-e6578830d3b6-kube-api-access-7z7z7\") pod \"distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.419238 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.419193    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:28.844702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:28.844676    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"]
Oct 31 19:46:29.727153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:29.727117    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" event={"ID":"e9f8518c-5924-44d3-8a07-e6578830d3b6","Type":"ContainerStarted","Data":"cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393"}
Oct 31 19:46:29.727153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:29.727161    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" event={"ID":"e9f8518c-5924-44d3-8a07-e6578830d3b6","Type":"ContainerStarted","Data":"5f108dda5980d5d2a541e012130fb5e92301d7432db9297b54e95a7b65c576f3"}
Oct 31 19:46:29.845230 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:29.845171    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podStartSLOduration=1.8451529469999999 podStartE2EDuration="1.845152947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:29.841130865 +0000 UTC m=+1808.370664920" watchObservedRunningTime="2026-10-31 19:46:29.845152947 +0000 UTC m=+1808.374687001"
Oct 31 19:46:30.290374 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:30.290325    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:30.419749 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:30.419706    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:33.466215 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:33.466169    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:34.510068 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.510025    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:46:34.646884 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.646846    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"]
Oct 31 19:46:34.651599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.651575    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:34.708049 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.707951    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"]
Oct 31 19:46:34.709031 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.709000    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:46:34.751008 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.750973    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:46:34.751185 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.751013    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:46:34.751185 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.751013    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:46:34.792446 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.792349    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5gb\" (UniqueName: \"kubernetes.io/projected/9dd4a721-1d07-4564-989e-c2b6a52e4fee-kube-api-access-5z5gb\") pod \"maas-api-revoke-sub-8434b597-qbb8k\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:34.792446 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.792394    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8434b597-qbb8k\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:34.792731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.792531    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8434b597-qbb8k\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:34.893832 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.893791    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8434b597-qbb8k\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:34.894054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.893944    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5gb\" (UniqueName: \"kubernetes.io/projected/9dd4a721-1d07-4564-989e-c2b6a52e4fee-kube-api-access-5z5gb\") pod \"maas-api-revoke-sub-8434b597-qbb8k\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:34.894054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.893970    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8434b597-qbb8k\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:34.894666 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.894633    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8434b597-qbb8k\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:34.896594 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.896555    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8434b597-qbb8k\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:34.952079 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.952041    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5gb\" (UniqueName: \"kubernetes.io/projected/9dd4a721-1d07-4564-989e-c2b6a52e4fee-kube-api-access-5z5gb\") pod \"maas-api-revoke-sub-8434b597-qbb8k\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:34.969299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:34.969254    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:35.270845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:35.270805    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:35.354717 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:46:35.354675    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd4a721_1d07_4564_989e_c2b6a52e4fee.slice/crio-c6ddaeab718ea27aa2f352c020bb058125f4e8c109a0bfd9fb1fd39a1bf394dd WatchSource:0}: Error finding container c6ddaeab718ea27aa2f352c020bb058125f4e8c109a0bfd9fb1fd39a1bf394dd: Status 404 returned error can't find the container with id c6ddaeab718ea27aa2f352c020bb058125f4e8c109a0bfd9fb1fd39a1bf394dd
Oct 31 19:46:35.359006 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:35.358982    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"]
Oct 31 19:46:35.408614 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:46:35.408540    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:46:35.789241 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:35.788460    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k" event={"ID":"9dd4a721-1d07-4564-989e-c2b6a52e4fee","Type":"ContainerStarted","Data":"c6ddaeab718ea27aa2f352c020bb058125f4e8c109a0bfd9fb1fd39a1bf394dd"}
Oct 31 19:46:36.799821 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:36.799780    5006 generic.go:356] "Generic (PLEG): container finished" podID="9dd4a721-1d07-4564-989e-c2b6a52e4fee" containerID="a02e1437c2f4eb778af8cfd45b1c9289a9c997fa2f8b8744ff17547918c34218" exitCode=0
Oct 31 19:46:36.800325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:36.799871    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k" event={"ID":"9dd4a721-1d07-4564-989e-c2b6a52e4fee","Type":"ContainerDied","Data":"a02e1437c2f4eb778af8cfd45b1c9289a9c997fa2f8b8744ff17547918c34218"}
Oct 31 19:46:38.006045 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.006015    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:38.144568 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.144529    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-xks-ca\") pod \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") "
Oct 31 19:46:38.144809 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.144623    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd4a721-1d07-4564-989e-c2b6a52e4fee-kube-api-access-5z5gb\" (UniqueName: \"kubernetes.io/projected/9dd4a721-1d07-4564-989e-c2b6a52e4fee-kube-api-access-5z5gb\") pod \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") "
Oct 31 19:46:38.144809 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.144751    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-openshift-ca\") pod \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\" (UID: \"9dd4a721-1d07-4564-989e-c2b6a52e4fee\") "
Oct 31 19:46:38.145495 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.145462    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-openshift-ca" pod "9dd4a721-1d07-4564-989e-c2b6a52e4fee" (UID: "9dd4a721-1d07-4564-989e-c2b6a52e4fee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:46:38.147119 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.147096    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd4a721-1d07-4564-989e-c2b6a52e4fee-kube-api-access-5z5gb" pod "9dd4a721-1d07-4564-989e-c2b6a52e4fee" (UID: "9dd4a721-1d07-4564-989e-c2b6a52e4fee"). InnerVolumeSpecName "kube-api-access-5z5gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:38.147298 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.147281    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-xks-ca" pod "9dd4a721-1d07-4564-989e-c2b6a52e4fee" (UID: "9dd4a721-1d07-4564-989e-c2b6a52e4fee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:46:38.246305 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.246261    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:38.246305 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.246296    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd4a721-1d07-4564-989e-c2b6a52e4fee-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:38.246305 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.246306    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z5gb\" (UniqueName: \"kubernetes.io/projected/9dd4a721-1d07-4564-989e-c2b6a52e4fee-kube-api-access-5z5gb\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:38.420354 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.420269    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:38.434048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.434016    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:38.792407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.792306    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:38.818364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.818334    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"
Oct 31 19:46:38.818522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.818372    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k" event={"ID":"9dd4a721-1d07-4564-989e-c2b6a52e4fee","Type":"ContainerDied","Data":"c6ddaeab718ea27aa2f352c020bb058125f4e8c109a0bfd9fb1fd39a1bf394dd"}
Oct 31 19:46:38.818522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.818422    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ddaeab718ea27aa2f352c020bb058125f4e8c109a0bfd9fb1fd39a1bf394dd"
Oct 31 19:46:38.832277 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:38.832247    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:46:40.274088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:40.274047    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:42.264160 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.264105    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"]
Oct 31 19:46:42.265444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.265416    5006 cpu_manager.go:414] "containerMap: removing container" podUID="9dd4a721-1d07-4564-989e-c2b6a52e4fee" containerName="revoke-keys"
Oct 31 19:46:42.265444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.265442    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd4a721-1d07-4564-989e-c2b6a52e4fee" containerName="revoke-keys"
Oct 31 19:46:42.265820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.265799    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd4a721-1d07-4564-989e-c2b6a52e4fee" containerName="revoke-keys"
Oct 31 19:46:42.270748 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.270717    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.385939 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.385892    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-967dee\"/\"kube-root-ca.crt\""
Oct 31 19:46:42.386153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.385961    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-967dee\"/\"openshift-service-ca.crt\""
Oct 31 19:46:42.386153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.385958    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-967dee\"/\"distinct-2-w2-967dee-kserve-self-signed-certs\""
Oct 31 19:46:42.386153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.386021    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-967dee\"/\"default-dockercfg-s96lf\""
Oct 31 19:46:42.390558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.390519    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"]
Oct 31 19:46:42.401503 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.401467    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-tmp-dir\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.401689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.401514    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9498e552-f1ed-4372-8da2-07058bbeaf58-tls-certs\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.401742 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.401635    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2shz\" (UniqueName: \"kubernetes.io/projected/9498e552-f1ed-4372-8da2-07058bbeaf58-kube-api-access-r2shz\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.401827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.401801    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-home\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.401966 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.401865    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-dshm\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.402045 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.402028    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-model-cache\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.487456 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.487410    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:42.503859 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.503821    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-tmp-dir\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.503859 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.503858    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9498e552-f1ed-4372-8da2-07058bbeaf58-tls-certs\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.504078 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.503994    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2shz\" (UniqueName: \"kubernetes.io/projected/9498e552-f1ed-4372-8da2-07058bbeaf58-kube-api-access-r2shz\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.504119 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.504089    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-home\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.504171 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.504159    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-dshm\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.504279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.504261    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-model-cache\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.504355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.504309    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-tmp-dir\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.504464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.504445    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-home\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.504811 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.504761    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-model-cache\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.506615 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.506590    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-dshm\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.507020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.506995    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9498e552-f1ed-4372-8da2-07058bbeaf58-tls-certs\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.562197 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.562115    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2shz\" (UniqueName: \"kubernetes.io/projected/9498e552-f1ed-4372-8da2-07058bbeaf58-kube-api-access-r2shz\") pod \"distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.585451 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.585405    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:42.970731 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:46:42.970696    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9498e552_f1ed_4372_8da2_07058bbeaf58.slice/crio-f40a5acf85e042e01328f7b1a575d4b43369fc1ff3417b46b54d4103b75dc669 WatchSource:0}: Error finding container f40a5acf85e042e01328f7b1a575d4b43369fc1ff3417b46b54d4103b75dc669: Status 404 returned error can't find the container with id f40a5acf85e042e01328f7b1a575d4b43369fc1ff3417b46b54d4103b75dc669
Oct 31 19:46:42.974796 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:42.974765    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"]
Oct 31 19:46:43.862889 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:43.862854    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" event={"ID":"9498e552-f1ed-4372-8da2-07058bbeaf58","Type":"ContainerStarted","Data":"50638469c3ff06f0fad4736738d7641635bde54cdcb906afb9977e80cce99715"}
Oct 31 19:46:43.863348 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:43.862897    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" event={"ID":"9498e552-f1ed-4372-8da2-07058bbeaf58","Type":"ContainerStarted","Data":"f40a5acf85e042e01328f7b1a575d4b43369fc1ff3417b46b54d4103b75dc669"}
Oct 31 19:46:43.982564 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:43.982500    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podStartSLOduration=1.9824831779999998 podStartE2EDuration="1.982483178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:43.97713211 +0000 UTC m=+1822.506666181" watchObservedRunningTime="2026-10-31 19:46:43.982483178 +0000 UTC m=+1822.512017233"
Oct 31 19:46:44.586416 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:44.586374    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:45.059333 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:45.059235    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"]
Oct 31 19:46:45.087885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:45.087844    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e62d352-9kxnj"]
Oct 31 19:46:45.181928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:45.181886    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:45.600050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:45.600015    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9dfab44-81de-4f36-9826-03f638684e49" path="/var/lib/kubelet/pods/f9dfab44-81de-4f36-9826-03f638684e49/volumes"
Oct 31 19:46:48.269845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.269800    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"]
Oct 31 19:46:48.273959 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.273932    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.326935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.326895    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-570915-openshift-default-dockercfg-87zft\""
Oct 31 19:46:48.381165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.381118    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.381395 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.381254    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.381395 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.381327    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.381531 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.381443    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.381531 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.381487    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x29t8\" (UniqueName: \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-kube-api-access-x29t8\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.381666 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.381580    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.381666 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.381618    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c660d7cd-b579-4885-a115-bee9024c128c-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.381815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.381787    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c660d7cd-b579-4885-a115-bee9024c128c-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.381876 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.381834    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.382198 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.382177    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"]
Oct 31 19:46:48.483472 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483437    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.483692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483495    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.483692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483526    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.483692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483555    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.483692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483583    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x29t8\" (UniqueName: \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-kube-api-access-x29t8\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.483692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483618    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.483934 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483695    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c660d7cd-b579-4885-a115-bee9024c128c-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.483934 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483801    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c660d7cd-b579-4885-a115-bee9024c128c-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.483934 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483844    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.484105 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483931    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.484105 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.483964    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.484453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.484431    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.484528 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.484456    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.484821 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.484799    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c660d7cd-b579-4885-a115-bee9024c128c-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.486106 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.486076    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.486447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.486427    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c660d7cd-b579-4885-a115-bee9024c128c-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.541290 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.541207    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.541461 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.541423    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x29t8\" (UniqueName: \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-kube-api-access-x29t8\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:48.589508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:48.589475    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:49.012785 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:46:49.012755    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc660d7cd_b579_4885_a115_bee9024c128c.slice/crio-4931e41b360bf29e30a4ac9ae88f9ecc08b4eaa50932d03dbfd979c7aba31d90 WatchSource:0}: Error finding container 4931e41b360bf29e30a4ac9ae88f9ecc08b4eaa50932d03dbfd979c7aba31d90: Status 404 returned error can't find the container with id 4931e41b360bf29e30a4ac9ae88f9ecc08b4eaa50932d03dbfd979c7aba31d90
Oct 31 19:46:49.015470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:49.015433    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:46:49.015588 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:49.015505    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:46:49.015588 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:49.015534    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:46:49.017585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:49.017559    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"]
Oct 31 19:46:49.931341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:49.931298    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" event={"ID":"c660d7cd-b579-4885-a115-bee9024c128c","Type":"ContainerStarted","Data":"cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6"}
Oct 31 19:46:49.931794 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:49.931366    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" event={"ID":"c660d7cd-b579-4885-a115-bee9024c128c","Type":"ContainerStarted","Data":"4931e41b360bf29e30a4ac9ae88f9ecc08b4eaa50932d03dbfd979c7aba31d90"}
Oct 31 19:46:50.049184 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:50.049151    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"]
Oct 31 19:46:50.053921 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:50.053889    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-879cc0ff-b6s64"]
Oct 31 19:46:50.068330 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:50.068285    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:50.128195 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:50.128121    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" podStartSLOduration=2.128097049 podStartE2EDuration="2.128097049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:50.116039214 +0000 UTC m=+1828.645573293" watchObservedRunningTime="2026-10-31 19:46:50.128097049 +0000 UTC m=+1828.657631097"
Oct 31 19:46:50.589920 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:50.589888    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:50.591771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:50.591745    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 19:46:50.591855 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:50.591792    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 19:46:51.590514 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:51.590473    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 19:46:51.590954 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:51.590525    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 19:46:51.624055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:51.624017    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f47dae-8f6d-481d-b818-6cd4d3980aee" path="/var/lib/kubelet/pods/31f47dae-8f6d-481d-b818-6cd4d3980aee/volumes"
Oct 31 19:46:52.586059 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:52.586017    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:52.590808 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:52.590771    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 19:46:52.591194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:52.590833    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 19:46:52.625258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:52.625219    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:52.966735 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:52.966599    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:53.591009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:53.590965    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 19:46:53.591428 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:53.591047    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 19:46:54.594008 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:54.593965    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 19:46:54.594624 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:54.594589    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 19:46:55.033312 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:55.033215    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:55.590961 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:55.590922    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 19:46:55.591278 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:55.591251    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 19:46:56.090666 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:56.089963    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"]
Oct 31 19:46:56.122667 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:56.118937    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8f47ae-2x9nq"]
Oct 31 19:46:56.399881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:56.399762    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:46:56.593911 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:56.593874    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 19:46:56.594207 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:56.594175    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 19:46:57.004824 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.004773    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:57.026793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.026748    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:46:57.027096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.027062    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:57.291481 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.290794    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:57.401877 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.401803    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:57.403256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.403230    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:46:57.706751 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.706703    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:46:57.729065 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.729012    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:57.736709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.731033    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:57.790584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.790540    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:46:57.790924 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.790892    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:57.802092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.793243    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:46:57.808464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.808417    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:57.873806 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.842243    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:57.876186 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.852944    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68908d9c-af34-4f37-9fd3-032f3e1e7535" path="/var/lib/kubelet/pods/68908d9c-af34-4f37-9fd3-032f3e1e7535/volumes"
Oct 31 19:46:57.876186 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.875918    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:46:57.879823 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.879780    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:57.906737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.906686    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"]
Oct 31 19:46:57.962838 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:57.962311    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:58.062028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.061986    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"]
Oct 31 19:46:58.062479 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.062458    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.063035 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.062994    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:46:58.082668 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.080266    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:58.098018 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.097978    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7192868-mvb7x\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.098474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.098446    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7192868-mvb7x\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.098690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.098669    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qx98\" (UniqueName: \"kubernetes.io/projected/07de6229-b7c1-47ec-af5a-14df75111488-kube-api-access-9qx98\") pod \"maas-api-revoke-sub-d7192868-mvb7x\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.135709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.133026    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:46:58.135709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.133190    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:46:58.135709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.133344    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:46:58.135709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.133423    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:46:58.201672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.200723    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7192868-mvb7x\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.201672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.200926    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7192868-mvb7x\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.201672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.200974    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qx98\" (UniqueName: \"kubernetes.io/projected/07de6229-b7c1-47ec-af5a-14df75111488-kube-api-access-9qx98\") pod \"maas-api-revoke-sub-d7192868-mvb7x\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.205672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.202279    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7192868-mvb7x\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.219737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.219577    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7192868-mvb7x\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.227379 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.227327    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:46:58.260300 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.260252    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qx98\" (UniqueName: \"kubernetes.io/projected/07de6229-b7c1-47ec-af5a-14df75111488-kube-api-access-9qx98\") pod \"maas-api-revoke-sub-d7192868-mvb7x\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.308133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.308097    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:58.309137 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.309102    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:58.310369 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.310342    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 19:46:58.445910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.445866    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:46:58.871198 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:46:58.871163    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07de6229_b7c1_47ec_af5a_14df75111488.slice/crio-9c17a8b6f9c89d56d7fc2a7d6bae804455a95f0e1a98107ce7f46a02f7f981e3 WatchSource:0}: Error finding container 9c17a8b6f9c89d56d7fc2a7d6bae804455a95f0e1a98107ce7f46a02f7f981e3: Status 404 returned error can't find the container with id 9c17a8b6f9c89d56d7fc2a7d6bae804455a95f0e1a98107ce7f46a02f7f981e3
Oct 31 19:46:58.877376 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.877347    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"]
Oct 31 19:46:58.915063 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.915012    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"]
Oct 31 19:46:58.920943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.920915    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:58.972926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:58.972870    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"]
Oct 31 19:46:59.017565 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.017494    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64972103-t8794\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:59.017827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.017631    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8npb\" (UniqueName: \"kubernetes.io/projected/dafcb30b-2516-46e0-a82b-9069cbb334b6-kube-api-access-j8npb\") pod \"maas-api-revoke-sub-64972103-t8794\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:59.017827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.017816    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64972103-t8794\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:59.118925 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.118822    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64972103-t8794\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:59.119088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.118981    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64972103-t8794\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:59.119088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.119033    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8npb\" (UniqueName: \"kubernetes.io/projected/dafcb30b-2516-46e0-a82b-9069cbb334b6-kube-api-access-j8npb\") pod \"maas-api-revoke-sub-64972103-t8794\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:59.119583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.119554    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64972103-t8794\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:59.121905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.121883    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64972103-t8794\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:59.155196 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.155161    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x" event={"ID":"07de6229-b7c1-47ec-af5a-14df75111488","Type":"ContainerStarted","Data":"9c17a8b6f9c89d56d7fc2a7d6bae804455a95f0e1a98107ce7f46a02f7f981e3"}
Oct 31 19:46:59.177091 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.177040    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8npb\" (UniqueName: \"kubernetes.io/projected/dafcb30b-2516-46e0-a82b-9069cbb334b6-kube-api-access-j8npb\") pod \"maas-api-revoke-sub-64972103-t8794\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:59.237027 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.236991    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:46:59.619309 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:46:59.619267    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafcb30b_2516_46e0_a82b_9069cbb334b6.slice/crio-c94edde9596c814a74e3dea9a6d4c20a2ab15150a9338b5c45808bd45926653d WatchSource:0}: Error finding container c94edde9596c814a74e3dea9a6d4c20a2ab15150a9338b5c45808bd45926653d: Status 404 returned error can't find the container with id c94edde9596c814a74e3dea9a6d4c20a2ab15150a9338b5c45808bd45926653d
Oct 31 19:46:59.625770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:46:59.625741    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"]
Oct 31 19:47:00.173453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:00.173341    5006 generic.go:356] "Generic (PLEG): container finished" podID="07de6229-b7c1-47ec-af5a-14df75111488" containerID="0548e306f6d9d9ddf506baebd37b9990c1410af8db9c6ccf18e6aa42fd631688" exitCode=0
Oct 31 19:47:00.173453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:00.173427    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x" event={"ID":"07de6229-b7c1-47ec-af5a-14df75111488","Type":"ContainerDied","Data":"0548e306f6d9d9ddf506baebd37b9990c1410af8db9c6ccf18e6aa42fd631688"}
Oct 31 19:47:00.175606 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:00.175569    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794" event={"ID":"dafcb30b-2516-46e0-a82b-9069cbb334b6","Type":"ContainerStarted","Data":"14ce114c50e6ec01edde8b35f4f28e0e634cbea1af45431c2d6b8b6bb5a7043f"}
Oct 31 19:47:00.175783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:00.175633    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794" event={"ID":"dafcb30b-2516-46e0-a82b-9069cbb334b6","Type":"ContainerStarted","Data":"c94edde9596c814a74e3dea9a6d4c20a2ab15150a9338b5c45808bd45926653d"}
Oct 31 19:47:00.401564 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:00.401491    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794" podStartSLOduration=2.401474364 podStartE2EDuration="2.401474364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:00.398394646 +0000 UTC m=+1838.927928699" watchObservedRunningTime="2026-10-31 19:47:00.401474364 +0000 UTC m=+1838.931008416"
Oct 31 19:47:00.480623 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:00.480538    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:01.185325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.185290    5006 generic.go:356] "Generic (PLEG): container finished" podID="dafcb30b-2516-46e0-a82b-9069cbb334b6" containerID="14ce114c50e6ec01edde8b35f4f28e0e634cbea1af45431c2d6b8b6bb5a7043f" exitCode=0
Oct 31 19:47:01.185815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.185379    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794" event={"ID":"dafcb30b-2516-46e0-a82b-9069cbb334b6","Type":"ContainerDied","Data":"14ce114c50e6ec01edde8b35f4f28e0e634cbea1af45431c2d6b8b6bb5a7043f"}
Oct 31 19:47:01.363424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.363396    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:47:01.448218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.448126    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-openshift-ca\") pod \"07de6229-b7c1-47ec-af5a-14df75111488\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") "
Oct 31 19:47:01.448218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.448176    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-xks-ca\") pod \"07de6229-b7c1-47ec-af5a-14df75111488\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") "
Oct 31 19:47:01.448218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.448208    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07de6229-b7c1-47ec-af5a-14df75111488-kube-api-access-9qx98\" (UniqueName: \"kubernetes.io/projected/07de6229-b7c1-47ec-af5a-14df75111488-kube-api-access-9qx98\") pod \"07de6229-b7c1-47ec-af5a-14df75111488\" (UID: \"07de6229-b7c1-47ec-af5a-14df75111488\") "
Oct 31 19:47:01.448552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.448509    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-openshift-ca" pod "07de6229-b7c1-47ec-af5a-14df75111488" (UID: "07de6229-b7c1-47ec-af5a-14df75111488"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:01.448817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.448793    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:01.450504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.450467    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07de6229-b7c1-47ec-af5a-14df75111488-kube-api-access-9qx98" pod "07de6229-b7c1-47ec-af5a-14df75111488" (UID: "07de6229-b7c1-47ec-af5a-14df75111488"). InnerVolumeSpecName "kube-api-access-9qx98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:01.450614 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.450501    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-xks-ca" pod "07de6229-b7c1-47ec-af5a-14df75111488" (UID: "07de6229-b7c1-47ec-af5a-14df75111488"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:01.550356 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.550308    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07de6229-b7c1-47ec-af5a-14df75111488-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:01.550356 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:01.550347    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qx98\" (UniqueName: \"kubernetes.io/projected/07de6229-b7c1-47ec-af5a-14df75111488-kube-api-access-9qx98\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:02.198354 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.198322    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"
Oct 31 19:47:02.198354 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.198347    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x" event={"ID":"07de6229-b7c1-47ec-af5a-14df75111488","Type":"ContainerDied","Data":"9c17a8b6f9c89d56d7fc2a7d6bae804455a95f0e1a98107ce7f46a02f7f981e3"}
Oct 31 19:47:02.198844 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.198392    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c17a8b6f9c89d56d7fc2a7d6bae804455a95f0e1a98107ce7f46a02f7f981e3"
Oct 31 19:47:02.376998 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.376968    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:47:02.461879 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.461789    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-xks-ca\") pod \"dafcb30b-2516-46e0-a82b-9069cbb334b6\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") "
Oct 31 19:47:02.462058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.461898    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafcb30b-2516-46e0-a82b-9069cbb334b6-kube-api-access-j8npb\" (UniqueName: \"kubernetes.io/projected/dafcb30b-2516-46e0-a82b-9069cbb334b6-kube-api-access-j8npb\") pod \"dafcb30b-2516-46e0-a82b-9069cbb334b6\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") "
Oct 31 19:47:02.462058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.461943    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-openshift-ca\") pod \"dafcb30b-2516-46e0-a82b-9069cbb334b6\" (UID: \"dafcb30b-2516-46e0-a82b-9069cbb334b6\") "
Oct 31 19:47:02.462338 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.462309    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-openshift-ca" pod "dafcb30b-2516-46e0-a82b-9069cbb334b6" (UID: "dafcb30b-2516-46e0-a82b-9069cbb334b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:02.463879 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.463857    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-xks-ca" pod "dafcb30b-2516-46e0-a82b-9069cbb334b6" (UID: "dafcb30b-2516-46e0-a82b-9069cbb334b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:02.464028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.464006    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafcb30b-2516-46e0-a82b-9069cbb334b6-kube-api-access-j8npb" pod "dafcb30b-2516-46e0-a82b-9069cbb334b6" (UID: "dafcb30b-2516-46e0-a82b-9069cbb334b6"). InnerVolumeSpecName "kube-api-access-j8npb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:02.564105 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.564061    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:02.564105 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.564100    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8npb\" (UniqueName: \"kubernetes.io/projected/dafcb30b-2516-46e0-a82b-9069cbb334b6-kube-api-access-j8npb\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:02.564105 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.564115    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dafcb30b-2516-46e0-a82b-9069cbb334b6-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:02.744421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.744335    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:02.992747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:02.992714    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:03.206578 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:03.206544    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"
Oct 31 19:47:03.207091 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:03.206582    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794" event={"ID":"dafcb30b-2516-46e0-a82b-9069cbb334b6","Type":"ContainerDied","Data":"c94edde9596c814a74e3dea9a6d4c20a2ab15150a9338b5c45808bd45926653d"}
Oct 31 19:47:03.207091 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:03.206631    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c94edde9596c814a74e3dea9a6d4c20a2ab15150a9338b5c45808bd45926653d"
Oct 31 19:47:05.395178 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:47:05.395138    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:47:05.469158 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:05.469121    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:09.637816 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.637770    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"]
Oct 31 19:47:09.638361 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.638344    5006 cpu_manager.go:414] "containerMap: removing container" podUID="dafcb30b-2516-46e0-a82b-9069cbb334b6" containerName="revoke-keys"
Oct 31 19:47:09.638361 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.638361    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafcb30b-2516-46e0-a82b-9069cbb334b6" containerName="revoke-keys"
Oct 31 19:47:09.638433 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.638394    5006 cpu_manager.go:414] "containerMap: removing container" podUID="07de6229-b7c1-47ec-af5a-14df75111488" containerName="revoke-keys"
Oct 31 19:47:09.638433 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.638399    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07de6229-b7c1-47ec-af5a-14df75111488" containerName="revoke-keys"
Oct 31 19:47:09.638505 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.638495    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="07de6229-b7c1-47ec-af5a-14df75111488" containerName="revoke-keys"
Oct 31 19:47:09.638542 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.638513    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafcb30b-2516-46e0-a82b-9069cbb334b6" containerName="revoke-keys"
Oct 31 19:47:09.642574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.642550    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:09.697319 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.697281    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"]
Oct 31 19:47:09.740474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.740438    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:47:09.740683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.740524    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:47:09.746331 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.746295    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:47:09.746331 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.746306    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:47:09.760153 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.760118    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeaf9b6d-hkctt\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:09.760315 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.760177    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc567\" (UniqueName: \"kubernetes.io/projected/eba86388-27ea-4e3a-8a4a-4556554dea7c-kube-api-access-hc567\") pod \"maas-api-revoke-sub-eeaf9b6d-hkctt\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:09.760407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.760384    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeaf9b6d-hkctt\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:09.861949 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.861907    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc567\" (UniqueName: \"kubernetes.io/projected/eba86388-27ea-4e3a-8a4a-4556554dea7c-kube-api-access-hc567\") pod \"maas-api-revoke-sub-eeaf9b6d-hkctt\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:09.862149 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.862085    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeaf9b6d-hkctt\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:09.862194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.862153    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeaf9b6d-hkctt\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:09.862805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.862776    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeaf9b6d-hkctt\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:09.864593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.864567    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeaf9b6d-hkctt\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:09.919304 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.919229    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc567\" (UniqueName: \"kubernetes.io/projected/eba86388-27ea-4e3a-8a4a-4556554dea7c-kube-api-access-hc567\") pod \"maas-api-revoke-sub-eeaf9b6d-hkctt\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:09.956850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:09.956810    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:10.063730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:10.061730    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"]
Oct 31 19:47:10.066576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:10.066521    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8a2f5d2-2nfvj"]
Oct 31 19:47:10.331247 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:10.331195    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba86388_27ea_4e3a_8a4a_4556554dea7c.slice/crio-545e153ffb7767a82c026f631d0b5ec16fb6692a9de67dd931580cfa9735076f WatchSource:0}: Error finding container 545e153ffb7767a82c026f631d0b5ec16fb6692a9de67dd931580cfa9735076f: Status 404 returned error can't find the container with id 545e153ffb7767a82c026f631d0b5ec16fb6692a9de67dd931580cfa9735076f
Oct 31 19:47:10.335316 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:10.335292    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"]
Oct 31 19:47:10.486385 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:10.486349    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:11.269222 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:11.269180    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt" event={"ID":"eba86388-27ea-4e3a-8a4a-4556554dea7c","Type":"ContainerStarted","Data":"f1c244bf4cd2d116c2eb1c7a9bac5ac8151370f23767a0085ccea48ee9431d63"}
Oct 31 19:47:11.269696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:11.269246    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt" event={"ID":"eba86388-27ea-4e3a-8a4a-4556554dea7c","Type":"ContainerStarted","Data":"545e153ffb7767a82c026f631d0b5ec16fb6692a9de67dd931580cfa9735076f"}
Oct 31 19:47:11.389559 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:11.389489    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt" podStartSLOduration=2.389472748 podStartE2EDuration="2.389472748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:11.3857761 +0000 UTC m=+1849.915310167" watchObservedRunningTime="2026-10-31 19:47:11.389472748 +0000 UTC m=+1849.919006845"
Oct 31 19:47:11.604736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:11.604700    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b7c9ad8-4428-40d2-a4bb-24dde49eab60" path="/var/lib/kubelet/pods/3b7c9ad8-4428-40d2-a4bb-24dde49eab60/volumes"
Oct 31 19:47:12.279536 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:12.279500    5006 generic.go:356] "Generic (PLEG): container finished" podID="eba86388-27ea-4e3a-8a4a-4556554dea7c" containerID="f1c244bf4cd2d116c2eb1c7a9bac5ac8151370f23767a0085ccea48ee9431d63" exitCode=0
Oct 31 19:47:12.279968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:12.279587    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt" event={"ID":"eba86388-27ea-4e3a-8a4a-4556554dea7c","Type":"ContainerDied","Data":"f1c244bf4cd2d116c2eb1c7a9bac5ac8151370f23767a0085ccea48ee9431d63"}
Oct 31 19:47:13.455764 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:13.455732    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:13.612947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:13.612912    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-openshift-ca\") pod \"eba86388-27ea-4e3a-8a4a-4556554dea7c\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") "
Oct 31 19:47:13.613158 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:13.613141    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-xks-ca\") pod \"eba86388-27ea-4e3a-8a4a-4556554dea7c\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") "
Oct 31 19:47:13.613257 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:13.613180    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba86388-27ea-4e3a-8a4a-4556554dea7c-kube-api-access-hc567\" (UniqueName: \"kubernetes.io/projected/eba86388-27ea-4e3a-8a4a-4556554dea7c-kube-api-access-hc567\") pod \"eba86388-27ea-4e3a-8a4a-4556554dea7c\" (UID: \"eba86388-27ea-4e3a-8a4a-4556554dea7c\") "
Oct 31 19:47:13.613415 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:13.613381    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-openshift-ca" pod "eba86388-27ea-4e3a-8a4a-4556554dea7c" (UID: "eba86388-27ea-4e3a-8a4a-4556554dea7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:13.613727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:13.613701    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:13.615954 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:13.615905    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba86388-27ea-4e3a-8a4a-4556554dea7c-kube-api-access-hc567" pod "eba86388-27ea-4e3a-8a4a-4556554dea7c" (UID: "eba86388-27ea-4e3a-8a4a-4556554dea7c"). InnerVolumeSpecName "kube-api-access-hc567". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:13.616178 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:13.616151    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-xks-ca" pod "eba86388-27ea-4e3a-8a4a-4556554dea7c" (UID: "eba86388-27ea-4e3a-8a4a-4556554dea7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:13.715362 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:13.715327    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eba86388-27ea-4e3a-8a4a-4556554dea7c-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:13.715362 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:13.715355    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc567\" (UniqueName: \"kubernetes.io/projected/eba86388-27ea-4e3a-8a4a-4556554dea7c-kube-api-access-hc567\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:14.296349 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:14.296306    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt" event={"ID":"eba86388-27ea-4e3a-8a4a-4556554dea7c","Type":"ContainerDied","Data":"545e153ffb7767a82c026f631d0b5ec16fb6692a9de67dd931580cfa9735076f"}
Oct 31 19:47:14.296349 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:14.296336    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"
Oct 31 19:47:14.296562 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:14.296368    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="545e153ffb7767a82c026f631d0b5ec16fb6692a9de67dd931580cfa9735076f"
Oct 31 19:47:15.386092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.386055    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:15.419043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.419005    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"]
Oct 31 19:47:15.419736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.419719    5006 cpu_manager.go:414] "containerMap: removing container" podUID="eba86388-27ea-4e3a-8a4a-4556554dea7c" containerName="revoke-keys"
Oct 31 19:47:15.419795 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.419738    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba86388-27ea-4e3a-8a4a-4556554dea7c" containerName="revoke-keys"
Oct 31 19:47:15.419870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.419859    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba86388-27ea-4e3a-8a4a-4556554dea7c" containerName="revoke-keys"
Oct 31 19:47:15.423673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.423628    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:15.477517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.477465    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"]
Oct 31 19:47:15.522476 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.522438    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:47:15.522700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.522479    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:47:15.522700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.522642    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:47:15.526690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.526664    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:47:15.538503 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.538464    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb77f644-4thbk\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:15.538605 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.538521    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bp4\" (UniqueName: \"kubernetes.io/projected/2287821a-df1a-4443-bbcc-207b2291fbeb-kube-api-access-m7bp4\") pod \"maas-api-revoke-sub-fb77f644-4thbk\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:15.538756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.538741    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb77f644-4thbk\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:15.640240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.640199    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb77f644-4thbk\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:15.640408 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.640301    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb77f644-4thbk\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:15.640408 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.640335    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bp4\" (UniqueName: \"kubernetes.io/projected/2287821a-df1a-4443-bbcc-207b2291fbeb-kube-api-access-m7bp4\") pod \"maas-api-revoke-sub-fb77f644-4thbk\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:15.640976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.640954    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb77f644-4thbk\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:15.642895 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.642874    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb77f644-4thbk\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:15.697789 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.697756    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bp4\" (UniqueName: \"kubernetes.io/projected/2287821a-df1a-4443-bbcc-207b2291fbeb-kube-api-access-m7bp4\") pod \"maas-api-revoke-sub-fb77f644-4thbk\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:15.738372 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:15.738325    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:16.112114 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:16.112080    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2287821a_df1a_4443_bbcc_207b2291fbeb.slice/crio-a7720cb1d41e461584b1a41bf53ae0c022bd5bc3fe8e00a619fa5d2341733635 WatchSource:0}: Error finding container a7720cb1d41e461584b1a41bf53ae0c022bd5bc3fe8e00a619fa5d2341733635: Status 404 returned error can't find the container with id a7720cb1d41e461584b1a41bf53ae0c022bd5bc3fe8e00a619fa5d2341733635
Oct 31 19:47:16.116526 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:16.116470    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"]
Oct 31 19:47:16.313731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:16.313696    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk" event={"ID":"2287821a-df1a-4443-bbcc-207b2291fbeb","Type":"ContainerStarted","Data":"a7720cb1d41e461584b1a41bf53ae0c022bd5bc3fe8e00a619fa5d2341733635"}
Oct 31 19:47:17.322416 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:17.322331    5006 generic.go:356] "Generic (PLEG): container finished" podID="2287821a-df1a-4443-bbcc-207b2291fbeb" containerID="b98d375fe76309b70b9b56020d28c680e416f0b45160f63853edcfcd8a0cbe58" exitCode=0
Oct 31 19:47:17.322822 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:17.322413    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk" event={"ID":"2287821a-df1a-4443-bbcc-207b2291fbeb","Type":"ContainerDied","Data":"b98d375fe76309b70b9b56020d28c680e416f0b45160f63853edcfcd8a0cbe58"}
Oct 31 19:47:18.215541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.215507    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"]
Oct 31 19:47:18.220128 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.220103    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.265916 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.265876    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"]
Oct 31 19:47:18.270425 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.270397    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.316618 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.316578    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"]
Oct 31 19:47:18.375378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.375329    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.375885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.375386    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.375885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.375464    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.375885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.375490    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nbk\" (UniqueName: \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-kube-api-access-h8nbk\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.375885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.375535    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.375885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.375596    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.377903    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.377949    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.377994    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.378145    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.378177    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.378202    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.378263    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.378296    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61fd271-fe36-43f7-a548-0675c01ef58a-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.378349    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.378387    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.378417    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrlw2\" (UniqueName: \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-kube-api-access-mrlw2\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.383917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.378442    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.386736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.386407    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"]
Oct 31 19:47:18.480261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480150    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.480261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480231    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.480261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480259    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.480602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480303    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.480602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480387    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.480602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480408    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.480602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480427    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.480602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480462    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.480602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480480    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61fd271-fe36-43f7-a548-0675c01ef58a-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.480602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480512    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.480602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480538    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.480602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480558    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrlw2\" (UniqueName: \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-kube-api-access-mrlw2\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.480602 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480574    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.481095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480679    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.481095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480713    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.481095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480741    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.481095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480759    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nbk\" (UniqueName: \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-kube-api-access-h8nbk\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.481095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480788    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.481095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480892    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.482771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.480707    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.482771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.481709    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61fd271-fe36-43f7-a548-0675c01ef58a-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.482771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.481749    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.482771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.482172    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.482771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.482419    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.482771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.482437    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.482771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.482621    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.482771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.482724    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.483284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.482845    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.485286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.485261    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.491717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.491685    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.491717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.491701    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.491962 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.491944    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.539169 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.539135    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.539339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.539176    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.539394 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.539369    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nbk\" (UniqueName: \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-kube-api-access-h8nbk\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.539506 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.539489    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrlw2\" (UniqueName: \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-kube-api-access-mrlw2\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.563496 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.563466    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:18.587661 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.587621    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:18.683617 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.683578    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-openshift-ca\") pod \"2287821a-df1a-4443-bbcc-207b2291fbeb\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") "
Oct 31 19:47:18.683845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.683771    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2287821a-df1a-4443-bbcc-207b2291fbeb-kube-api-access-m7bp4\" (UniqueName: \"kubernetes.io/projected/2287821a-df1a-4443-bbcc-207b2291fbeb-kube-api-access-m7bp4\") pod \"2287821a-df1a-4443-bbcc-207b2291fbeb\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") "
Oct 31 19:47:18.683845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.683824    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-xks-ca\") pod \"2287821a-df1a-4443-bbcc-207b2291fbeb\" (UID: \"2287821a-df1a-4443-bbcc-207b2291fbeb\") "
Oct 31 19:47:18.684710 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.684327    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-openshift-ca" pod "2287821a-df1a-4443-bbcc-207b2291fbeb" (UID: "2287821a-df1a-4443-bbcc-207b2291fbeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:18.684710 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.684498    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:18.686853 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.686821    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2287821a-df1a-4443-bbcc-207b2291fbeb-kube-api-access-m7bp4" pod "2287821a-df1a-4443-bbcc-207b2291fbeb" (UID: "2287821a-df1a-4443-bbcc-207b2291fbeb"). InnerVolumeSpecName "kube-api-access-m7bp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:18.687039 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.686853    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-xks-ca" pod "2287821a-df1a-4443-bbcc-207b2291fbeb" (UID: "2287821a-df1a-4443-bbcc-207b2291fbeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:18.786347 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.786254    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7bp4\" (UniqueName: \"kubernetes.io/projected/2287821a-df1a-4443-bbcc-207b2291fbeb-kube-api-access-m7bp4\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:18.786347 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.786290    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2287821a-df1a-4443-bbcc-207b2291fbeb-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:18.818400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.818362    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"]
Oct 31 19:47:18.819074 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.819055    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2287821a-df1a-4443-bbcc-207b2291fbeb" containerName="revoke-keys"
Oct 31 19:47:18.819143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.819076    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2287821a-df1a-4443-bbcc-207b2291fbeb" containerName="revoke-keys"
Oct 31 19:47:18.819219 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.819207    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2287821a-df1a-4443-bbcc-207b2291fbeb" containerName="revoke-keys"
Oct 31 19:47:18.824053 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.824027    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.834223 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.834194    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:18.881838 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.881797    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"]
Oct 31 19:47:18.891277 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.891232    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.891444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.891318    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbk8f\" (UniqueName: \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-kube-api-access-sbk8f\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.891444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.891386    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.891444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.891428    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.891612 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.891454    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.891612 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.891498    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.891612 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.891555    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.891793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.891637    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.891793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.891703    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.992686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.992631    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993007 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.992753    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993007 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.992825    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbk8f\" (UniqueName: \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-kube-api-access-sbk8f\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993007 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.992899    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993188 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.993015    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993188 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.993062    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993188 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.993117    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993188 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.993142    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993188 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.993181    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993452 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.993200    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993452 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.993299    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993452 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.993392    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.993630 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.993493    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.994369 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.994343    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.995864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.995843    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:18.996259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:18.996238    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:19.008724 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:19.008685    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61fd271_fe36_43f7_a548_0675c01ef58a.slice/crio-fc7a6330645b80a677e05acf35d0c3cadbf9a57245d968265e91beddd8a02332 WatchSource:0}: Error finding container fc7a6330645b80a677e05acf35d0c3cadbf9a57245d968265e91beddd8a02332: Status 404 returned error can't find the container with id fc7a6330645b80a677e05acf35d0c3cadbf9a57245d968265e91beddd8a02332
Oct 31 19:47:19.011460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.011419    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:19.011582 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.011489    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:19.011582 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.011521    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:19.011809 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.011783    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"]
Oct 31 19:47:19.051142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.051058    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:19.051293 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.051161    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbk8f\" (UniqueName: \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-kube-api-access-sbk8f\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:19.139283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.139239    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:19.174289 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.174247    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"]
Oct 31 19:47:19.180266 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.180233    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.197857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.197155    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f49887-fe0c-440a-bdee-a73a16e56385-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.197857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.197364    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.197857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.197405    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.197857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.197504    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.197857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.197543    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f49887-fe0c-440a-bdee-a73a16e56385-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.197857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.197599    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.197857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.197767    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.197857 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.197845    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.198419 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.197967    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84prx\" (UniqueName: \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-kube-api-access-84prx\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.231156 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.230496    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"]
Oct 31 19:47:19.254880 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.244542    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:19.254880 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.244626    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:19.254880 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.244733    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:19.260449 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.258967    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"]
Oct 31 19:47:19.299928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.299888    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84prx\" (UniqueName: \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-kube-api-access-84prx\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.300095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.299972    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f49887-fe0c-440a-bdee-a73a16e56385-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.300095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.300083    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.300178 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.300116    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.300178 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.300150    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.300248 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.300183    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f49887-fe0c-440a-bdee-a73a16e56385-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.300291 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.300251    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.300326 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.300286    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.300368 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.300334    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.300959 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.300894    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.301415 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.301338    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.302203 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.302175    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f49887-fe0c-440a-bdee-a73a16e56385-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.302472 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.302448    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.302780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.302757    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.305958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.305924    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f49887-fe0c-440a-bdee-a73a16e56385-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.307946 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.307919    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.342600 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.342563    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" event={"ID":"f4ea5406-4c0a-4faf-8e91-230b2e442e63","Type":"ContainerStarted","Data":"0141675fd0aca0658c4a9704e122b1392be2436f2b31c61c11368bc47fe64dea"}
Oct 31 19:47:19.345371 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.345168    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" event={"ID":"d61fd271-fe36-43f7-a548-0675c01ef58a","Type":"ContainerStarted","Data":"227d63be116789257900d498490fafcd26862c6e32a20bacaa9a59aa5084c2bc"}
Oct 31 19:47:19.345371 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.345238    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" event={"ID":"d61fd271-fe36-43f7-a548-0675c01ef58a","Type":"ContainerStarted","Data":"fc7a6330645b80a677e05acf35d0c3cadbf9a57245d968265e91beddd8a02332"}
Oct 31 19:47:19.347492 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.347460    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk" event={"ID":"2287821a-df1a-4443-bbcc-207b2291fbeb","Type":"ContainerDied","Data":"a7720cb1d41e461584b1a41bf53ae0c022bd5bc3fe8e00a619fa5d2341733635"}
Oct 31 19:47:19.347698 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.347555    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7720cb1d41e461584b1a41bf53ae0c022bd5bc3fe8e00a619fa5d2341733635"
Oct 31 19:47:19.347698 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.347621    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"
Oct 31 19:47:19.358885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.358853    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84prx\" (UniqueName: \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-kube-api-access-84prx\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.360102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.360072    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-67h59\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.469111 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.469034    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podStartSLOduration=1.469015649 podStartE2EDuration="1.469015649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:19.466887599 +0000 UTC m=+1857.996421652" watchObservedRunningTime="2026-10-31 19:47:19.469015649 +0000 UTC m=+1857.998549702"
Oct 31 19:47:19.480815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.480772    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"]
Oct 31 19:47:19.486426 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.486397    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.499440 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.499403    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:19.554418 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.544326    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"]
Oct 31 19:47:19.554418 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.552554    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"]
Oct 31 19:47:19.608632 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.605405    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:19.608632 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.605586    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:19.608632 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.605703    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:19.608922 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.608761    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fadbb9-e442-4672-af4e-735d47f90808-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.608922 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.608817    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.609055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.609027    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66v4w\" (UniqueName: \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-kube-api-access-66v4w\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.609112 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.609099    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.609842 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.609227    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fadbb9-e442-4672-af4e-735d47f90808-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.609842 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.609273    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.609842 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.609303    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.609842 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.609368    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.609842 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.609399    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.622035 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.622003    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:19.710616 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.710582    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fadbb9-e442-4672-af4e-735d47f90808-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.710906 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.710883    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.711040 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.711024    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.711184 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.711167    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.711291 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.711278    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.711469 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.711442    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.711547 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.711459    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fadbb9-e442-4672-af4e-735d47f90808-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.711681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.711663    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.711925 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.711907    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66v4w\" (UniqueName: \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-kube-api-access-66v4w\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.712088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.712072    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.712268 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.712137    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.712388 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.712291    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.712619 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.712597    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fadbb9-e442-4672-af4e-735d47f90808-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.712905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.712882    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.714921 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.714843    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.717463 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.717433    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fadbb9-e442-4672-af4e-735d47f90808-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.770038 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.770005    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.770303 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.770276    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66v4w\" (UniqueName: \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-kube-api-access-66v4w\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:19.911562 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:19.911517    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:20.013457 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:20.013416    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f49887_fe0c_440a_bdee_a73a16e56385.slice/crio-5f2edd4b3459c2537e7a9acca3b95975e73b75837a4d7f031dd722b216f6f2a9 WatchSource:0}: Error finding container 5f2edd4b3459c2537e7a9acca3b95975e73b75837a4d7f031dd722b216f6f2a9: Status 404 returned error can't find the container with id 5f2edd4b3459c2537e7a9acca3b95975e73b75837a4d7f031dd722b216f6f2a9
Oct 31 19:47:20.015203 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.015169    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"]
Oct 31 19:47:20.018042 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.017429    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:20.018042 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.017526    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:20.018042 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.017582    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:20.350679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.342022    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"]
Oct 31 19:47:20.374975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.374924    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:20.375146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.375028    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:20.375146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.375069    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:20.379833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.379160    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" event={"ID":"f2fadbb9-e442-4672-af4e-735d47f90808","Type":"ContainerStarted","Data":"ae62cad914ba1f01b9315d79063de30c460d895cd15498607d208fd0715dd205"}
Oct 31 19:47:20.383243 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.383208    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" event={"ID":"92f49887-fe0c-440a-bdee-a73a16e56385","Type":"ContainerStarted","Data":"97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3"}
Oct 31 19:47:20.383383 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.383276    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" event={"ID":"92f49887-fe0c-440a-bdee-a73a16e56385","Type":"ContainerStarted","Data":"5f2edd4b3459c2537e7a9acca3b95975e73b75837a4d7f031dd722b216f6f2a9"}
Oct 31 19:47:20.386711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.386639    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" event={"ID":"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262","Type":"ContainerStarted","Data":"51b6872224cf2f254b460d1fb025ef9dfcc7bce84f0ba4a5bc3010ce7db519f1"}
Oct 31 19:47:20.386871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.386738    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" event={"ID":"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262","Type":"ContainerStarted","Data":"eb1b4921f31c8d4b11c87de94aaa02dc6edf073db9d5ab811899a612013b05aa"}
Oct 31 19:47:20.391092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.391047    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" event={"ID":"f4ea5406-4c0a-4faf-8e91-230b2e442e63","Type":"ContainerStarted","Data":"18cdf54dcf17a807d7cce0cc3d64f2d7b86b42c4b58b7e15924f6c542e0469a7"}
Oct 31 19:47:20.462763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.461572    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:20.502067 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.499782    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:20.575502 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.575433    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podStartSLOduration=1.575395795 podStartE2EDuration="1.575395795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:20.516971475 +0000 UTC m=+1859.046505555" watchObservedRunningTime="2026-10-31 19:47:20.575395795 +0000 UTC m=+1859.104929853"
Oct 31 19:47:20.589026 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.588918    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 31 19:47:20.589026 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.588966    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 31 19:47:20.672207 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.672132    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podStartSLOduration=2.672022771 podStartE2EDuration="2.672022771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:20.639194726 +0000 UTC m=+1859.168728816" watchObservedRunningTime="2026-10-31 19:47:20.672022771 +0000 UTC m=+1859.201556822"
Oct 31 19:47:20.773677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.768349    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podStartSLOduration=2.768325232 podStartE2EDuration="2.768325232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:20.761856031 +0000 UTC m=+1859.291390086" watchObservedRunningTime="2026-10-31 19:47:20.768325232 +0000 UTC m=+1859.297859287"
Oct 31 19:47:20.834951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.834853    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:20.838378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.837515    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:47:20.838692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:20.838641    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 19:47:21.107801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.107764    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"]
Oct 31 19:47:21.167675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.167236    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:21.167675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.167415    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:47:21.167675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.167464    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 31 19:47:21.190677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.188514    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd24f59-bmp9v"]
Oct 31 19:47:21.446737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.444916    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" event={"ID":"f2fadbb9-e442-4672-af4e-735d47f90808","Type":"ContainerStarted","Data":"4eff85c45043d92a0e3cf39b4bae291992657214ffe5d1d36be4b7480804d9ea"}
Oct 31 19:47:21.516808 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.516769    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:21.545798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.545752    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:21.591673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.589567    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 31 19:47:21.591673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.589629    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 31 19:47:21.700770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.697367    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podStartSLOduration=2.697342787 podStartE2EDuration="2.697342787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:21.575111565 +0000 UTC m=+1860.104645649" watchObservedRunningTime="2026-10-31 19:47:21.697342787 +0000 UTC m=+1860.226876841"
Oct 31 19:47:21.758727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.758686    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae2de8c-1742-488e-a1f7-140acbe32eab" path="/var/lib/kubelet/pods/eae2de8c-1742-488e-a1f7-140acbe32eab/volumes"
Oct 31 19:47:21.857674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.841815    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:47:21.857674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.841878    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 19:47:21.916673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.916071    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:21.916673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.916213    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:47:21.916673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:21.916256    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 19:47:22.059924 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.059798    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:22.070567 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.070528    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:22.137861 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.137795    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:22.139845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.139816    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:47:22.140103 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.140068    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 31 19:47:22.262579 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.262532    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:22.274183 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.262860    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:22.447673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.446113    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:22.506689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.503142    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:22.506689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.503205    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:22.531676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.520776    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:22.591674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.591569    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:47:22.595679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.594528    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 31 19:47:22.595679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.594575    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 31 19:47:22.708753 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.708192    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:47:22.723803 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.723744    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:22.838673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.838083    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:47:22.838673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.838145    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 19:47:22.845476 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.845431    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:22.847159 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.846747    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:22.847576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.846783    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:47:22.912302 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.912253    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:47:22.912586 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:22.912561    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 19:47:23.142130 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.142090    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:47:23.142373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.142347    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 31 19:47:23.144052 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.142541    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:47:23.144052 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.142908    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:47:23.194805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.194752    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:23.468733 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.456616    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:23.468733 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.457054    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:47:23.472673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.469050    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:23.472673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.470905    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:47:23.579676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.547223    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:23.579676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.547409    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:23.579676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.578224    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:23.579676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.578331    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:47:23.623429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.623376    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:23.849982 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.849921    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:23.877675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.865996    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:47:23.923222 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.870429    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:23.924413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.887568    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 31 19:47:23.924598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.924565    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 31 19:47:23.924760 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.887771    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:47:23.942713 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.931807    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:47:23.942713 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.931866    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 19:47:23.942713 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.932403    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:47:23.942713 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.932444    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 19:47:23.995162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.995129    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:47:23.995579 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.995560    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:47:23.997293 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:23.995859    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:47:24.000882 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.000862    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:47:24.045705 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.020880    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:47:24.045705 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.022737    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.071638 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.071579    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.102892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.102785    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.232678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.219790    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:47:24.232678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.219852    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 31 19:47:24.278180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.278137    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.351003 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.185643    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.493864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.493475    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.523677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.494125    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 31 19:47:24.567683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.567622    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded"
Oct 31 19:47:24.732936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.605759    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:24.733558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.733511    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:24.733725 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.635221    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.733835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.627201    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.734253 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.674198    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 31 19:47:24.734422 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.734399    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 31 19:47:24.865679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.850433    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.923196 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.904504    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:47:24.923424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.923398    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 19:47:24.923520 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.922854    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.923905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.905203    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:47:24.924340 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.924310    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:24.947724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.926844    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:47:24.967209 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.967166    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:47:24.967384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:24.967225    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 19:47:25.035679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.023481    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:25.048913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.048877    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:47:25.108314 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.108182    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:25.108731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.108703    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 31 19:47:25.116260 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.116176    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:47:25.152878 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.152828    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:47:25.206874 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.206831    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 31 19:47:25.370870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.370764    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:25.371988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.371961    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:47:25.374493 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.374464    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:47:25.525693 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.496736    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:25.526703 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.526663    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:25.619678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.616051    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:25.645499 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.645398    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:25.645777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.619927    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 31 19:47:25.645931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.645908    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 31 19:47:25.781708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.756440    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:25.830730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.776795    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:47:25.884676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.869742    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:25.884676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.872074    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:47:25.884676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.872135    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 19:47:25.934740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.924008    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:25.934740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.926254    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 31 19:47:25.935148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.935117    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:47:25.935316 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.935290    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 19:47:25.936585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.936552    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:25.950793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.950728    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:25.967382 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:25.967302    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:26.109145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.109112    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:47:26.109446 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.109417    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:26.124181 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.124135    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:26.200735 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.171854    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:47:26.259813 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.204724    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 31 19:47:26.259813 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.206359    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:47:26.330786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.293799    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:26.425673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.424730    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:47:26.425673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.425124    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:47:26.436890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.436839    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:26.550731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.508420    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:47:26.550731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.510355    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:26.550731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.510403    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:26.628728 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.551865    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:26.656675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.630823    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 31 19:47:26.657005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.656965    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 31 19:47:26.762676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.742942    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:26.791900 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.791748    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:26.824384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.824269    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 31 19:47:26.825523 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.825480    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:26.845714 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.845676    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:47:26.849769 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.849717    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 19:47:27.140675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.931023    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:47:27.140906 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.140745    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 19:47:27.140906 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.944085    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:27.173687 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.943846    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:27.173687 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.980407    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:27.323675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.303994    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:47:27.323675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.304078    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 31 19:47:27.323675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.126928    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:27.323675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.126859    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:27.323675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:26.978980    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:47:27.483364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.439094    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:27.540171 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.540129    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:47:27.753772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.551984    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:27.753772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.552738    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:27.753772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.552786    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:27.753772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.629096    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 31 19:47:27.753772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.629157    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 31 19:47:27.753772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.636034    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:27.850013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.849965    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:27.903676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.849974    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:47:27.932154 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.861237    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:47:27.932437 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.932401    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 19:47:27.961929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:27.961878    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:47:28.031277 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.031175    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 19:47:28.053416 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.032383    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:28.152675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.151754    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:28.152675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.152412    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:28.204677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.204368    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:28.554117 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.552077    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:47:28.554117 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.552406    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:47:28.556191 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.556161    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:47:28.578937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.578891    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 31 19:47:28.579220 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.578836    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:28.579373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.579350    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:28.580392 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.580362    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:47:28.581768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.580374    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:28.614396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.589234    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:47:28.614396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.590002    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 31 19:47:28.614396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.590062    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 31 19:47:28.657674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.647090    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:28.842824 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:47:28.842788    5006 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.229s"
Oct 31 19:47:28.849015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.848982    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:47:28.849253 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.849227    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 19:47:28.911888 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.911828    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 31 19:47:28.913339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.913222    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:47:28.913644 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.913617    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 19:47:28.936845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:28.936789    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:47:29.036741 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.036693    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:47:29.082941 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.082884    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:29.090236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.090197    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:47:29.192729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.173591    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:29.192729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.173984    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 31 19:47:29.228674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.217280    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:29.425701 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.315040    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:47:29.493781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.447243    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"]
Oct 31 19:47:29.506046 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.487751    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 31 19:47:29.536676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.512763    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:29.575673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.566948    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:29.575673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.567020    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:29.648701 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.624815    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:47:29.659634 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.649270    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:47:29.718674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.663372    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:47:29.873860 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.665963    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:47:29.877678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.665998    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 31 19:47:29.881976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.666031    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:29.885455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.666058    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:29.890459 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.874556    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:47:29.895846 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.894779    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 19:47:29.931406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.931360    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 31 19:47:29.952640 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.934776    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:47:29.952640 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.951903    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 19:47:29.952640 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:29.937271    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:30.105678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.099065    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:47:30.111339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.111315    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:47:30.111575 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.111562    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:47:30.120050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.115171    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:30.120488 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.120468    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:47:30.146619 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.146536    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:47:30.147913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.147886    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:47:30.148105 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.148079    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 31 19:47:30.203675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.197751    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:30.203675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.198146    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:47:30.203675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.200452    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:30.204068 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.204045    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:30.232674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.230946    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 19:47:30.257676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.250483    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeb43b24-xxx2d"]
Oct 31 19:47:30.439722 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.439509    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:30.529473 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.526773    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:30.529473 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.527284    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:47:30.533826 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.533786    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:30.534070 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.534038    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:30.655827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.655781    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:30.728785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.728688    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:30.923081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.922849    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 31 19:47:30.958917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.958868    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:47:30.961598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.961566    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:47:30.961852 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:30.961822    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 19:47:31.037160 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.037051    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:31.297741 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.293108    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:31.501689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.487754    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:31.520814 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.499131    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:47:31.552867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.552766    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-67h59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:47:31.631786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.631746    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 19:47:31.632073 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.586002    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 31 19:47:31.824211 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.723079    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Oct 31 19:47:31.824467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.790803    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:31.882686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.848778    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:31.942676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.887876    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:31.998738 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:31.985752    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:32.126845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.099262    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:47:32.126845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.119167    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:47:32.126845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.121957    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 31 19:47:32.196923 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.196865    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:47:32.197207 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.197177    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:32.266677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.234435    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0605535-ff87-4ce9-84b0-b40eb9cdce3f" path="/var/lib/kubelet/pods/e0605535-ff87-4ce9-84b0-b40eb9cdce3f/volumes"
Oct 31 19:47:32.266677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.235060    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:47:32.266677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.235137    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:47:32.266677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.235258    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:32.266677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.235352    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:47:32.266677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.243301    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:32.302676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.268790    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 19:47:32.434729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.429190    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:32.434729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.431518    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:32.458058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.444152    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:47:32.473237 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.470637    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:47:32.515566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.513858    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:47:32.515566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.514236    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:47:32.524692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.523790    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:47:32.919811 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.919762    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 31 19:47:32.929490 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.929456    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:47:32.938996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.938955    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:47:32.940678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:32.939833    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:47:33.131670 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.102724    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:47:33.131670 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.104909    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:47:33.131670 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.126069    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:33.139009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.136529    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 31 19:47:33.139009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.136873    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:47:33.139009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.137166    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:33.139009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.137414    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:47:33.139009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.137450    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:33.139426 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.139251    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:33.140994 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.140626    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 19:47:33.162148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.159248    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:33.197163 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.194800    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:47:33.249807 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.249751    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:33.291232 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.291205    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:47:33.373596 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.373307    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:33.377804 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.377779    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:33.382072 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.381923    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:47:33.417874 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.413988    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:47:33.450740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.450189    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:47:33.527051 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.526739    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:33.551681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.549493    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:47:33.563923 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.563896    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:47:33.571986 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.571958    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:47:33.614170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:33.614089    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:47:35.049717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.049679    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"]
Oct 31 19:47:35.054310 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.054279    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d72ef74b-tjbrg"]
Oct 31 19:47:35.227855 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.227823    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"]
Oct 31 19:47:35.233939 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.233913    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.286988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.286954    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:47:35.286988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.286968    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:47:35.286988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.286990    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:47:35.287288 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.287032    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:47:35.287755 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.287733    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"]
Oct 31 19:47:35.354474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.354435    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8tqr\" (UniqueName: \"kubernetes.io/projected/9a152abe-7297-4cda-9402-f1e48690a136-kube-api-access-f8tqr\") pod \"maas-api-revoke-sub-763de9ca-p6sxv\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.354718 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.354574    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-763de9ca-p6sxv\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.354718 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.354641    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-763de9ca-p6sxv\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.394883 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:47:35.394845    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:47:35.456597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.456539    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8tqr\" (UniqueName: \"kubernetes.io/projected/9a152abe-7297-4cda-9402-f1e48690a136-kube-api-access-f8tqr\") pod \"maas-api-revoke-sub-763de9ca-p6sxv\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.457199 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.457170    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-763de9ca-p6sxv\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.457331 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.457213    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-763de9ca-p6sxv\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.458107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.458081    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-763de9ca-p6sxv\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.460013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.459992    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-763de9ca-p6sxv\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.515054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.515022    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8tqr\" (UniqueName: \"kubernetes.io/projected/9a152abe-7297-4cda-9402-f1e48690a136-kube-api-access-f8tqr\") pod \"maas-api-revoke-sub-763de9ca-p6sxv\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.550025 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.549984    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:35.602318 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.602278    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09967626-8fc7-4427-a695-8c1dbc2493ad" path="/var/lib/kubelet/pods/09967626-8fc7-4427-a695-8c1dbc2493ad/volumes"
Oct 31 19:47:35.935632 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:35.935600    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a152abe_7297_4cda_9402_f1e48690a136.slice/crio-1ef8b7edce732b0234510e6725aa387277f1876077f427c9978dfa4fab9ed272 WatchSource:0}: Error finding container 1ef8b7edce732b0234510e6725aa387277f1876077f427c9978dfa4fab9ed272: Status 404 returned error can't find the container with id 1ef8b7edce732b0234510e6725aa387277f1876077f427c9978dfa4fab9ed272
Oct 31 19:47:35.950081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:35.947110    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"]
Oct 31 19:47:36.054591 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:36.054559    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"]
Oct 31 19:47:36.058662 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:36.058625    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a9185c0-46cnw"]
Oct 31 19:47:36.349388 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:36.349350    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv" event={"ID":"9a152abe-7297-4cda-9402-f1e48690a136","Type":"ContainerStarted","Data":"1ef8b7edce732b0234510e6725aa387277f1876077f427c9978dfa4fab9ed272"}
Oct 31 19:47:36.351370 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:36.351328    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:37.052551 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.052513    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"]
Oct 31 19:47:37.056555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.056530    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae8856a-gw7cm"]
Oct 31 19:47:37.359556 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.359519    5006 generic.go:356] "Generic (PLEG): container finished" podID="9a152abe-7297-4cda-9402-f1e48690a136" containerID="6a78df03d79b043208e8c5eb44bd282282091d403efe81dd1331ff364131ce25" exitCode=0
Oct 31 19:47:37.359794 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.359607    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv" event={"ID":"9a152abe-7297-4cda-9402-f1e48690a136","Type":"ContainerDied","Data":"6a78df03d79b043208e8c5eb44bd282282091d403efe81dd1331ff364131ce25"}
Oct 31 19:47:37.495827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.495793    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"]
Oct 31 19:47:37.500199 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.500178    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:37.552313 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.552280    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"]
Oct 31 19:47:37.596836 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.596796    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfb98d45-jk2nl\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:37.597045 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.596890    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfb98d45-jk2nl\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:37.597117 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.597030    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5bk4\" (UniqueName: \"kubernetes.io/projected/b22f7cee-3ce0-4387-ac41-318b473c6794-kube-api-access-f5bk4\") pod \"maas-api-revoke-sub-cfb98d45-jk2nl\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:37.603642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.603601    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c55589-c14a-4908-8bde-6cb47f40da3a" path="/var/lib/kubelet/pods/58c55589-c14a-4908-8bde-6cb47f40da3a/volumes"
Oct 31 19:47:37.604292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.604268    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7796af89-4ef9-450a-8b69-2cc56820df8c" path="/var/lib/kubelet/pods/7796af89-4ef9-450a-8b69-2cc56820df8c/volumes"
Oct 31 19:47:37.698873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.698774    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfb98d45-jk2nl\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:37.698873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.698856    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5bk4\" (UniqueName: \"kubernetes.io/projected/b22f7cee-3ce0-4387-ac41-318b473c6794-kube-api-access-f5bk4\") pod \"maas-api-revoke-sub-cfb98d45-jk2nl\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:37.699120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.699049    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfb98d45-jk2nl\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:37.699513 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.699494    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfb98d45-jk2nl\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:37.701583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.701559    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfb98d45-jk2nl\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:37.759837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.759806    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5bk4\" (UniqueName: \"kubernetes.io/projected/b22f7cee-3ce0-4387-ac41-318b473c6794-kube-api-access-f5bk4\") pod \"maas-api-revoke-sub-cfb98d45-jk2nl\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:37.814796 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:37.814755    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:38.197900 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:38.197853    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22f7cee_3ce0_4387_ac41_318b473c6794.slice/crio-a12d6436d08ff6793cd8a4e65108ad0c518d5389ed3a2e6709ea5ccb72f4f40e WatchSource:0}: Error finding container a12d6436d08ff6793cd8a4e65108ad0c518d5389ed3a2e6709ea5ccb72f4f40e: Status 404 returned error can't find the container with id a12d6436d08ff6793cd8a4e65108ad0c518d5389ed3a2e6709ea5ccb72f4f40e
Oct 31 19:47:38.204835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.204810    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"]
Oct 31 19:47:38.370074 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.370037    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl" event={"ID":"b22f7cee-3ce0-4387-ac41-318b473c6794","Type":"ContainerStarted","Data":"a12d6436d08ff6793cd8a4e65108ad0c518d5389ed3a2e6709ea5ccb72f4f40e"}
Oct 31 19:47:38.425225 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.425190    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"]
Oct 31 19:47:38.431163 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.431136    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:38.484856 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.484772    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"]
Oct 31 19:47:38.508968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.508931    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c89baa59-fv29q\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:38.509104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.509059    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm4rs\" (UniqueName: \"kubernetes.io/projected/7ddacb46-c32c-421c-8f7d-6f44adfba25d-kube-api-access-sm4rs\") pod \"maas-api-revoke-sub-c89baa59-fv29q\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:38.509241 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.509221    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c89baa59-fv29q\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:38.514815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.514792    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:38.610324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.610297    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-openshift-ca\") pod \"9a152abe-7297-4cda-9402-f1e48690a136\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") "
Oct 31 19:47:38.610439 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.610362    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a152abe-7297-4cda-9402-f1e48690a136-kube-api-access-f8tqr\" (UniqueName: \"kubernetes.io/projected/9a152abe-7297-4cda-9402-f1e48690a136-kube-api-access-f8tqr\") pod \"9a152abe-7297-4cda-9402-f1e48690a136\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") "
Oct 31 19:47:38.610439 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.610401    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-xks-ca\") pod \"9a152abe-7297-4cda-9402-f1e48690a136\" (UID: \"9a152abe-7297-4cda-9402-f1e48690a136\") "
Oct 31 19:47:38.610675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.610639    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm4rs\" (UniqueName: \"kubernetes.io/projected/7ddacb46-c32c-421c-8f7d-6f44adfba25d-kube-api-access-sm4rs\") pod \"maas-api-revoke-sub-c89baa59-fv29q\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:38.610769 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.610643    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-openshift-ca" pod "9a152abe-7297-4cda-9402-f1e48690a136" (UID: "9a152abe-7297-4cda-9402-f1e48690a136"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:38.610868 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.610849    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c89baa59-fv29q\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:38.611103 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.611081    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c89baa59-fv29q\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:38.611209 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.611194    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:38.611766 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.611740    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c89baa59-fv29q\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:38.613735 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.613715    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c89baa59-fv29q\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:38.613735 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.613722    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a152abe-7297-4cda-9402-f1e48690a136-kube-api-access-f8tqr" pod "9a152abe-7297-4cda-9402-f1e48690a136" (UID: "9a152abe-7297-4cda-9402-f1e48690a136"). InnerVolumeSpecName "kube-api-access-f8tqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:38.613864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.613734    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-xks-ca" pod "9a152abe-7297-4cda-9402-f1e48690a136" (UID: "9a152abe-7297-4cda-9402-f1e48690a136"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:38.668451 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.668419    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm4rs\" (UniqueName: \"kubernetes.io/projected/7ddacb46-c32c-421c-8f7d-6f44adfba25d-kube-api-access-sm4rs\") pod \"maas-api-revoke-sub-c89baa59-fv29q\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:38.713071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.713035    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8tqr\" (UniqueName: \"kubernetes.io/projected/9a152abe-7297-4cda-9402-f1e48690a136-kube-api-access-f8tqr\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:38.713071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.713068    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a152abe-7297-4cda-9402-f1e48690a136-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:38.754763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:38.754694    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:39.133903 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:39.133869    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"]
Oct 31 19:47:39.151206 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:39.151174    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddacb46_c32c_421c_8f7d_6f44adfba25d.slice/crio-e707ebae4670d5465bfc35402d17821f1cb3adfdb7bedf0bf2073782e2be7686 WatchSource:0}: Error finding container e707ebae4670d5465bfc35402d17821f1cb3adfdb7bedf0bf2073782e2be7686: Status 404 returned error can't find the container with id e707ebae4670d5465bfc35402d17821f1cb3adfdb7bedf0bf2073782e2be7686
Oct 31 19:47:39.379744 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:39.379641    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q" event={"ID":"7ddacb46-c32c-421c-8f7d-6f44adfba25d","Type":"ContainerStarted","Data":"e707ebae4670d5465bfc35402d17821f1cb3adfdb7bedf0bf2073782e2be7686"}
Oct 31 19:47:39.381417 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:39.381387    5006 generic.go:356] "Generic (PLEG): container finished" podID="b22f7cee-3ce0-4387-ac41-318b473c6794" containerID="307bf2e37be7013104ca25042c7087cb9964261ab0a1e494d0aa7bc41ff83a27" exitCode=0
Oct 31 19:47:39.381562 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:39.381480    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl" event={"ID":"b22f7cee-3ce0-4387-ac41-318b473c6794","Type":"ContainerDied","Data":"307bf2e37be7013104ca25042c7087cb9964261ab0a1e494d0aa7bc41ff83a27"}
Oct 31 19:47:39.383256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:39.383237    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"
Oct 31 19:47:39.383373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:39.383278    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv" event={"ID":"9a152abe-7297-4cda-9402-f1e48690a136","Type":"ContainerDied","Data":"1ef8b7edce732b0234510e6725aa387277f1876077f427c9978dfa4fab9ed272"}
Oct 31 19:47:39.383373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:39.383318    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ef8b7edce732b0234510e6725aa387277f1876077f427c9978dfa4fab9ed272"
Oct 31 19:47:40.403191 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.403155    5006 generic.go:356] "Generic (PLEG): container finished" podID="7ddacb46-c32c-421c-8f7d-6f44adfba25d" containerID="c3a4309a0552002ee784dcf44896cb318b00662105a2084bf490fad2896ffedb" exitCode=0
Oct 31 19:47:40.403677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.403234    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q" event={"ID":"7ddacb46-c32c-421c-8f7d-6f44adfba25d","Type":"ContainerDied","Data":"c3a4309a0552002ee784dcf44896cb318b00662105a2084bf490fad2896ffedb"}
Oct 31 19:47:40.581843 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.581797    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"]
Oct 31 19:47:40.582550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.582523    5006 cpu_manager.go:414] "containerMap: removing container" podUID="9a152abe-7297-4cda-9402-f1e48690a136" containerName="revoke-keys"
Oct 31 19:47:40.582550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.582543    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a152abe-7297-4cda-9402-f1e48690a136" containerName="revoke-keys"
Oct 31 19:47:40.582781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.582765    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a152abe-7297-4cda-9402-f1e48690a136" containerName="revoke-keys"
Oct 31 19:47:40.586861 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.586841    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:40.590131 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.590098    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:40.642700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.639366    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"]
Oct 31 19:47:40.742906 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.742814    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22f7cee-3ce0-4387-ac41-318b473c6794-kube-api-access-f5bk4\" (UniqueName: \"kubernetes.io/projected/b22f7cee-3ce0-4387-ac41-318b473c6794-kube-api-access-f5bk4\") pod \"b22f7cee-3ce0-4387-ac41-318b473c6794\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") "
Oct 31 19:47:40.743084 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.742982    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-xks-ca\") pod \"b22f7cee-3ce0-4387-ac41-318b473c6794\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") "
Oct 31 19:47:40.743084 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.743032    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-openshift-ca\") pod \"b22f7cee-3ce0-4387-ac41-318b473c6794\" (UID: \"b22f7cee-3ce0-4387-ac41-318b473c6794\") "
Oct 31 19:47:40.743213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.743172    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-817f0b73-8jqhr\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:40.743213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.743195    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-817f0b73-8jqhr\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:40.743450 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.743426    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-openshift-ca" pod "b22f7cee-3ce0-4387-ac41-318b473c6794" (UID: "b22f7cee-3ce0-4387-ac41-318b473c6794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:40.743534 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.743447    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk9km\" (UniqueName: \"kubernetes.io/projected/42070ba2-fbe1-4b38-afc6-2cc3003006b1-kube-api-access-rk9km\") pod \"maas-api-revoke-sub-817f0b73-8jqhr\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:40.743683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.743644    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:40.745092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.745070    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22f7cee-3ce0-4387-ac41-318b473c6794-kube-api-access-f5bk4" pod "b22f7cee-3ce0-4387-ac41-318b473c6794" (UID: "b22f7cee-3ce0-4387-ac41-318b473c6794"). InnerVolumeSpecName "kube-api-access-f5bk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:40.745172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.745135    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-xks-ca" pod "b22f7cee-3ce0-4387-ac41-318b473c6794" (UID: "b22f7cee-3ce0-4387-ac41-318b473c6794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:40.844958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.844913    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk9km\" (UniqueName: \"kubernetes.io/projected/42070ba2-fbe1-4b38-afc6-2cc3003006b1-kube-api-access-rk9km\") pod \"maas-api-revoke-sub-817f0b73-8jqhr\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:40.845132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.845103    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-817f0b73-8jqhr\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:40.845224 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.845132    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-817f0b73-8jqhr\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:40.845337 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.845318    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5bk4\" (UniqueName: \"kubernetes.io/projected/b22f7cee-3ce0-4387-ac41-318b473c6794-kube-api-access-f5bk4\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:40.845392 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.845344    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22f7cee-3ce0-4387-ac41-318b473c6794-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:40.846087 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.846055    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-817f0b73-8jqhr\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:40.847906 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.847879    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-817f0b73-8jqhr\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:40.902334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:40.902305    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk9km\" (UniqueName: \"kubernetes.io/projected/42070ba2-fbe1-4b38-afc6-2cc3003006b1-kube-api-access-rk9km\") pod \"maas-api-revoke-sub-817f0b73-8jqhr\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:41.201584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.201545    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:41.414483 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.414448    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"
Oct 31 19:47:41.414952 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.414482    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl" event={"ID":"b22f7cee-3ce0-4387-ac41-318b473c6794","Type":"ContainerDied","Data":"a12d6436d08ff6793cd8a4e65108ad0c518d5389ed3a2e6709ea5ccb72f4f40e"}
Oct 31 19:47:41.414952 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.414532    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a12d6436d08ff6793cd8a4e65108ad0c518d5389ed3a2e6709ea5ccb72f4f40e"
Oct 31 19:47:41.456445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.456356    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:41.588056 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.588003    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"]
Oct 31 19:47:41.604592 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.604568    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:41.671396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.671364    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-openshift-ca\") pod \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") "
Oct 31 19:47:41.671530 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.671431    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ddacb46-c32c-421c-8f7d-6f44adfba25d-kube-api-access-sm4rs\" (UniqueName: \"kubernetes.io/projected/7ddacb46-c32c-421c-8f7d-6f44adfba25d-kube-api-access-sm4rs\") pod \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") "
Oct 31 19:47:41.671593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.671575    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-xks-ca\") pod \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\" (UID: \"7ddacb46-c32c-421c-8f7d-6f44adfba25d\") "
Oct 31 19:47:41.671836 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.671814    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-openshift-ca" pod "7ddacb46-c32c-421c-8f7d-6f44adfba25d" (UID: "7ddacb46-c32c-421c-8f7d-6f44adfba25d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:41.672148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.672124    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:41.673574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.673553    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-xks-ca" pod "7ddacb46-c32c-421c-8f7d-6f44adfba25d" (UID: "7ddacb46-c32c-421c-8f7d-6f44adfba25d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:41.673695 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.673579    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ddacb46-c32c-421c-8f7d-6f44adfba25d-kube-api-access-sm4rs" pod "7ddacb46-c32c-421c-8f7d-6f44adfba25d" (UID: "7ddacb46-c32c-421c-8f7d-6f44adfba25d"). InnerVolumeSpecName "kube-api-access-sm4rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:41.773477 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.773435    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm4rs\" (UniqueName: \"kubernetes.io/projected/7ddacb46-c32c-421c-8f7d-6f44adfba25d-kube-api-access-sm4rs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:41.773477 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.773471    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ddacb46-c32c-421c-8f7d-6f44adfba25d-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:41.970899 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.970863    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"]
Oct 31 19:47:41.971730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.971704    5006 cpu_manager.go:414] "containerMap: removing container" podUID="7ddacb46-c32c-421c-8f7d-6f44adfba25d" containerName="revoke-keys"
Oct 31 19:47:41.971730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.971729    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddacb46-c32c-421c-8f7d-6f44adfba25d" containerName="revoke-keys"
Oct 31 19:47:41.971995 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.971804    5006 cpu_manager.go:414] "containerMap: removing container" podUID="b22f7cee-3ce0-4387-ac41-318b473c6794" containerName="revoke-keys"
Oct 31 19:47:41.971995 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.971811    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22f7cee-3ce0-4387-ac41-318b473c6794" containerName="revoke-keys"
Oct 31 19:47:41.971995 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.971940    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddacb46-c32c-421c-8f7d-6f44adfba25d" containerName="revoke-keys"
Oct 31 19:47:41.971995 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.971961    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22f7cee-3ce0-4387-ac41-318b473c6794" containerName="revoke-keys"
Oct 31 19:47:41.975589 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:41.975568    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.031941 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.031848    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"]
Oct 31 19:47:42.076955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.076914    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3beeba99-xncgm\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.076955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.076953    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3beeba99-xncgm\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.077177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.077129    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s6cj\" (UniqueName: \"kubernetes.io/projected/271a1724-3bee-4943-9639-90f9cf8233bb-kube-api-access-9s6cj\") pod \"maas-api-revoke-sub-3beeba99-xncgm\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.178908 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.178871    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s6cj\" (UniqueName: \"kubernetes.io/projected/271a1724-3bee-4943-9639-90f9cf8233bb-kube-api-access-9s6cj\") pod \"maas-api-revoke-sub-3beeba99-xncgm\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.179086 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.179009    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3beeba99-xncgm\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.179086 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.179037    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3beeba99-xncgm\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.179690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.179645    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3beeba99-xncgm\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.181425 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.181406    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3beeba99-xncgm\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.236208 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.236173    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s6cj\" (UniqueName: \"kubernetes.io/projected/271a1724-3bee-4943-9639-90f9cf8233bb-kube-api-access-9s6cj\") pod \"maas-api-revoke-sub-3beeba99-xncgm\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.289358 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.289283    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:42.429691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.429633    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"
Oct 31 19:47:42.430149 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.429630    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q" event={"ID":"7ddacb46-c32c-421c-8f7d-6f44adfba25d","Type":"ContainerDied","Data":"e707ebae4670d5465bfc35402d17821f1cb3adfdb7bedf0bf2073782e2be7686"}
Oct 31 19:47:42.430149 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.429830    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e707ebae4670d5465bfc35402d17821f1cb3adfdb7bedf0bf2073782e2be7686"
Oct 31 19:47:42.431838 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.431812    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr" event={"ID":"42070ba2-fbe1-4b38-afc6-2cc3003006b1","Type":"ContainerStarted","Data":"681c4ffc47ee251b834ed5299b2c4c77551d66f7d40a567a6c530a3dd532c43e"}
Oct 31 19:47:42.431961 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.431865    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr" event={"ID":"42070ba2-fbe1-4b38-afc6-2cc3003006b1","Type":"ContainerStarted","Data":"dea1a84dcfe4d985d4982da25e1750134cb60fa7ad66c4d30e8c375d543028cf"}
Oct 31 19:47:42.546079 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.545983    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr" podStartSLOduration=2.5459705169999998 podStartE2EDuration="2.545970517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:42.545505487 +0000 UTC m=+1881.075039557" watchObservedRunningTime="2026-10-31 19:47:42.545970517 +0000 UTC m=+1881.075504612"
Oct 31 19:47:42.666868 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:42.666838    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271a1724_3bee_4943_9639_90f9cf8233bb.slice/crio-03c6014f1cdb6f472fe15ddeec5dcfce8e9d21faa198644377208cf705bbb41a WatchSource:0}: Error finding container 03c6014f1cdb6f472fe15ddeec5dcfce8e9d21faa198644377208cf705bbb41a: Status 404 returned error can't find the container with id 03c6014f1cdb6f472fe15ddeec5dcfce8e9d21faa198644377208cf705bbb41a
Oct 31 19:47:42.671558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:42.671526    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"]
Oct 31 19:47:43.441807 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:43.441769    5006 generic.go:356] "Generic (PLEG): container finished" podID="42070ba2-fbe1-4b38-afc6-2cc3003006b1" containerID="681c4ffc47ee251b834ed5299b2c4c77551d66f7d40a567a6c530a3dd532c43e" exitCode=0
Oct 31 19:47:43.442275 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:43.441853    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr" event={"ID":"42070ba2-fbe1-4b38-afc6-2cc3003006b1","Type":"ContainerDied","Data":"681c4ffc47ee251b834ed5299b2c4c77551d66f7d40a567a6c530a3dd532c43e"}
Oct 31 19:47:43.443697 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:43.443667    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm" event={"ID":"271a1724-3bee-4943-9639-90f9cf8233bb","Type":"ContainerStarted","Data":"b98b5f5f506a94f6dcc8cec32889afac06a2f30043c4ea09539de66f48783b7f"}
Oct 31 19:47:43.443830 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:43.443709    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm" event={"ID":"271a1724-3bee-4943-9639-90f9cf8233bb","Type":"ContainerStarted","Data":"03c6014f1cdb6f472fe15ddeec5dcfce8e9d21faa198644377208cf705bbb41a"}
Oct 31 19:47:43.673254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:43.673200    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm" podStartSLOduration=2.673184706 podStartE2EDuration="2.673184706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:43.668581052 +0000 UTC m=+1882.198115105" watchObservedRunningTime="2026-10-31 19:47:43.673184706 +0000 UTC m=+1882.202718758"
Oct 31 19:47:44.453621 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.453586    5006 generic.go:356] "Generic (PLEG): container finished" podID="271a1724-3bee-4943-9639-90f9cf8233bb" containerID="b98b5f5f506a94f6dcc8cec32889afac06a2f30043c4ea09539de66f48783b7f" exitCode=0
Oct 31 19:47:44.454086 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.453685    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm" event={"ID":"271a1724-3bee-4943-9639-90f9cf8233bb","Type":"ContainerDied","Data":"b98b5f5f506a94f6dcc8cec32889afac06a2f30043c4ea09539de66f48783b7f"}
Oct 31 19:47:44.637508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.637479    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:44.713464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.713377    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-xks-ca\") pod \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") "
Oct 31 19:47:44.713625 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.713489    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-openshift-ca\") pod \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") "
Oct 31 19:47:44.713714 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.713623    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42070ba2-fbe1-4b38-afc6-2cc3003006b1-kube-api-access-rk9km\" (UniqueName: \"kubernetes.io/projected/42070ba2-fbe1-4b38-afc6-2cc3003006b1-kube-api-access-rk9km\") pod \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\" (UID: \"42070ba2-fbe1-4b38-afc6-2cc3003006b1\") "
Oct 31 19:47:44.713906 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.713870    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-openshift-ca" pod "42070ba2-fbe1-4b38-afc6-2cc3003006b1" (UID: "42070ba2-fbe1-4b38-afc6-2cc3003006b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:44.714341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.714322    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:44.715624 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.715591    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-xks-ca" pod "42070ba2-fbe1-4b38-afc6-2cc3003006b1" (UID: "42070ba2-fbe1-4b38-afc6-2cc3003006b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:44.715819 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.715793    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42070ba2-fbe1-4b38-afc6-2cc3003006b1-kube-api-access-rk9km" pod "42070ba2-fbe1-4b38-afc6-2cc3003006b1" (UID: "42070ba2-fbe1-4b38-afc6-2cc3003006b1"). InnerVolumeSpecName "kube-api-access-rk9km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:44.815389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.815352    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk9km\" (UniqueName: \"kubernetes.io/projected/42070ba2-fbe1-4b38-afc6-2cc3003006b1-kube-api-access-rk9km\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:44.815389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:44.815384    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42070ba2-fbe1-4b38-afc6-2cc3003006b1-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:45.463098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.463066    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"
Oct 31 19:47:45.463466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.463104    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr" event={"ID":"42070ba2-fbe1-4b38-afc6-2cc3003006b1","Type":"ContainerDied","Data":"dea1a84dcfe4d985d4982da25e1750134cb60fa7ad66c4d30e8c375d543028cf"}
Oct 31 19:47:45.463466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.463154    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea1a84dcfe4d985d4982da25e1750134cb60fa7ad66c4d30e8c375d543028cf"
Oct 31 19:47:45.652520 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.652489    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:45.727605 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.727524    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271a1724-3bee-4943-9639-90f9cf8233bb-kube-api-access-9s6cj\" (UniqueName: \"kubernetes.io/projected/271a1724-3bee-4943-9639-90f9cf8233bb-kube-api-access-9s6cj\") pod \"271a1724-3bee-4943-9639-90f9cf8233bb\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") "
Oct 31 19:47:45.727790 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.727734    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-openshift-ca\") pod \"271a1724-3bee-4943-9639-90f9cf8233bb\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") "
Oct 31 19:47:45.727847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.727829    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-xks-ca\") pod \"271a1724-3bee-4943-9639-90f9cf8233bb\" (UID: \"271a1724-3bee-4943-9639-90f9cf8233bb\") "
Oct 31 19:47:45.728133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.728109    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-openshift-ca" pod "271a1724-3bee-4943-9639-90f9cf8233bb" (UID: "271a1724-3bee-4943-9639-90f9cf8233bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:45.728342 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.728324    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:45.729758 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.729729    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271a1724-3bee-4943-9639-90f9cf8233bb-kube-api-access-9s6cj" pod "271a1724-3bee-4943-9639-90f9cf8233bb" (UID: "271a1724-3bee-4943-9639-90f9cf8233bb"). InnerVolumeSpecName "kube-api-access-9s6cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:45.729849 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.729802    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-xks-ca" pod "271a1724-3bee-4943-9639-90f9cf8233bb" (UID: "271a1724-3bee-4943-9639-90f9cf8233bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:45.829756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.829721    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s6cj\" (UniqueName: \"kubernetes.io/projected/271a1724-3bee-4943-9639-90f9cf8233bb-kube-api-access-9s6cj\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:45.829756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:45.829752    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271a1724-3bee-4943-9639-90f9cf8233bb-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:46.356132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.356087    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:46.423116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.423060    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"]
Oct 31 19:47:46.424760 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.424731    5006 cpu_manager.go:414] "containerMap: removing container" podUID="42070ba2-fbe1-4b38-afc6-2cc3003006b1" containerName="revoke-keys"
Oct 31 19:47:46.424897 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.424825    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42070ba2-fbe1-4b38-afc6-2cc3003006b1" containerName="revoke-keys"
Oct 31 19:47:46.424897 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.424852    5006 cpu_manager.go:414] "containerMap: removing container" podUID="271a1724-3bee-4943-9639-90f9cf8233bb" containerName="revoke-keys"
Oct 31 19:47:46.424897 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.424872    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271a1724-3bee-4943-9639-90f9cf8233bb" containerName="revoke-keys"
Oct 31 19:47:46.425558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.425113    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="42070ba2-fbe1-4b38-afc6-2cc3003006b1" containerName="revoke-keys"
Oct 31 19:47:46.425558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.425142    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="271a1724-3bee-4943-9639-90f9cf8233bb" containerName="revoke-keys"
Oct 31 19:47:46.429396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.429374    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:46.474407 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.474364    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"
Oct 31 19:47:46.474937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.474410    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm" event={"ID":"271a1724-3bee-4943-9639-90f9cf8233bb","Type":"ContainerDied","Data":"03c6014f1cdb6f472fe15ddeec5dcfce8e9d21faa198644377208cf705bbb41a"}
Oct 31 19:47:46.474937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.474471    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03c6014f1cdb6f472fe15ddeec5dcfce8e9d21faa198644377208cf705bbb41a"
Oct 31 19:47:46.478216 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.478185    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"]
Oct 31 19:47:46.541151 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.541114    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a46f39-79p7r\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:46.541374 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.541353    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a46f39-79p7r\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:46.541415 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.541389    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtp82\" (UniqueName: \"kubernetes.io/projected/ec52aa5f-4450-4549-828a-ac1b3ae29b39-kube-api-access-jtp82\") pod \"maas-api-revoke-sub-65a46f39-79p7r\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:46.643070 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.642979    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a46f39-79p7r\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:46.643070 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.643020    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtp82\" (UniqueName: \"kubernetes.io/projected/ec52aa5f-4450-4549-828a-ac1b3ae29b39-kube-api-access-jtp82\") pod \"maas-api-revoke-sub-65a46f39-79p7r\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:46.643288 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.643119    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a46f39-79p7r\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:46.643832 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.643803    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a46f39-79p7r\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:46.645660 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.645624    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a46f39-79p7r\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:46.700554 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.700516    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtp82\" (UniqueName: \"kubernetes.io/projected/ec52aa5f-4450-4549-828a-ac1b3ae29b39-kube-api-access-jtp82\") pod \"maas-api-revoke-sub-65a46f39-79p7r\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:46.746486 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:46.746451    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:47.128812 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:47.128777    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec52aa5f_4450_4549_828a_ac1b3ae29b39.slice/crio-b1972fb08c342ebcf3766b78e08e88e8ae5f221e03027cd14535e14443a82ccb WatchSource:0}: Error finding container b1972fb08c342ebcf3766b78e08e88e8ae5f221e03027cd14535e14443a82ccb: Status 404 returned error can't find the container with id b1972fb08c342ebcf3766b78e08e88e8ae5f221e03027cd14535e14443a82ccb
Oct 31 19:47:47.134599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:47.134577    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"]
Oct 31 19:47:47.483639 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:47.483552    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r" event={"ID":"ec52aa5f-4450-4549-828a-ac1b3ae29b39","Type":"ContainerStarted","Data":"b1972fb08c342ebcf3766b78e08e88e8ae5f221e03027cd14535e14443a82ccb"}
Oct 31 19:47:48.051970 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.051934    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"]
Oct 31 19:47:48.102364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.102322    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-316dc445-mwzdd"]
Oct 31 19:47:48.220743 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.220706    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"]
Oct 31 19:47:48.225158 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.225137    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.267425 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.267390    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.267606 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.267471    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.267606 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.267552    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.267606 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.267583    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.267731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.267675    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.267784 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.267770    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.267816 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.267798    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rl8\" (UniqueName: \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-kube-api-access-h9rl8\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.267924 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.267901    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.267956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.267935    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.279882 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.279850    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"]
Oct 31 19:47:48.369032 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.368990    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369224 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369150    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369224 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369179    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rl8\" (UniqueName: \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-kube-api-access-h9rl8\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369235    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369336    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369424    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369497 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369438    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369567 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369549    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369628 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369616    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369696    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369794 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369729    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.369973 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.369954    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.370072 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.370052    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.370411 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.370384    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.371813 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.371785    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.372052 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.372033    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.426707 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.426674    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rl8\" (UniqueName: \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-kube-api-access-h9rl8\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.426859 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.426798    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.493224 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.493186    5006 generic.go:356] "Generic (PLEG): container finished" podID="ec52aa5f-4450-4549-828a-ac1b3ae29b39" containerID="a2ea4fba815c9a592d78e7bb5e687d36dda651761329aa04633994614acb242c" exitCode=0
Oct 31 19:47:48.493641 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.493262    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r" event={"ID":"ec52aa5f-4450-4549-828a-ac1b3ae29b39","Type":"ContainerDied","Data":"a2ea4fba815c9a592d78e7bb5e687d36dda651761329aa04633994614acb242c"}
Oct 31 19:47:48.539208 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.539170    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:48.772306 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.772217    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"]
Oct 31 19:47:48.778324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.778299    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.798593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.798550    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"]
Oct 31 19:47:48.803054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.803025    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.834072 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.834039    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"]
Oct 31 19:47:48.878729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.878641    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.878960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.878741    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.878960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.878794    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.878960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.878852    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.878960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.878900    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.878960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.878956    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.879249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879017    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222bdfa6-5c87-47d5-9091-c60527060166-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.879249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879084    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.879249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879125    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.879249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879146    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222bdfa6-5c87-47d5-9091-c60527060166-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.879249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879177    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjsmv\" (UniqueName: \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-kube-api-access-mjsmv\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.879249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879207    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.879419 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879268    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.879419 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879328    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.879419 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879359    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.879419 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879411    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.879534 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879447    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbdv\" (UniqueName: \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-kube-api-access-6sbdv\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.879534 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.879507    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.888362 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.888304    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"]
Oct 31 19:47:48.924090 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:48.924037    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6312c0d6_8021_4810_a98c_04aab3ad1e8e.slice/crio-1653072b01fcad9f46fa89fae3f75c8e30238eab6ef8ef3e3310f58619c78c3b WatchSource:0}: Error finding container 1653072b01fcad9f46fa89fae3f75c8e30238eab6ef8ef3e3310f58619c78c3b: Status 404 returned error can't find the container with id 1653072b01fcad9f46fa89fae3f75c8e30238eab6ef8ef3e3310f58619c78c3b
Oct 31 19:47:48.929642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.929607    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"]
Oct 31 19:47:48.932217 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.932175    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:48.932293 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.932265    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:48.932343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.932314    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:48.981071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981038    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.981213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981078    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.981213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981099    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.981213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981126    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.981213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981162    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.981213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981210    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.981485 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981274    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222bdfa6-5c87-47d5-9091-c60527060166-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.981485 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981349    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.981485 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981404    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.981485 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981429    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222bdfa6-5c87-47d5-9091-c60527060166-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.981485 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981460    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjsmv\" (UniqueName: \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-kube-api-access-mjsmv\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.981786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981487    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.981786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981552    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.981786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981610    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.981786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981642    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.981786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981707    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.981786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981740    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbdv\" (UniqueName: \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-kube-api-access-6sbdv\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.982158 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981796    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.982158 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.981810    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.982158 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.982113    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.982307 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.982191    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.982307 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.982257    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222bdfa6-5c87-47d5-9091-c60527060166-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.982807 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.982510    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.982807 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.982575    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.982807 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.982712    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.983058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.982843    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.983058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.982935    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.983058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.982969    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.983873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.983850    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.984661 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.984628    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222bdfa6-5c87-47d5-9091-c60527060166-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:48.984833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.984815    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:48.985234 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:48.985207    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:49.039332 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.039293    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:49.039509 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.039424    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjsmv\" (UniqueName: \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-kube-api-access-mjsmv\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:49.039818 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.039793    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:49.039923 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.039842    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbdv\" (UniqueName: \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-kube-api-access-6sbdv\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:49.105324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.093691    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:49.118245 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.118211    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:49.493277 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:49.493242    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a278f1d_df5d_40be_9b77_9292e1c8bc55.slice/crio-0e06e5920eeb4a0b3ef877bf05d417094757d9efbcdb756e2360319028079367 WatchSource:0}: Error finding container 0e06e5920eeb4a0b3ef877bf05d417094757d9efbcdb756e2360319028079367: Status 404 returned error can't find the container with id 0e06e5920eeb4a0b3ef877bf05d417094757d9efbcdb756e2360319028079367
Oct 31 19:47:49.496414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.496368    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:49.496575 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.496456    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:49.496575 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.496497    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:49.498572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.498548    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"]
Oct 31 19:47:49.504176 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.504147    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" event={"ID":"6312c0d6-8021-4810-a98c-04aab3ad1e8e","Type":"ContainerStarted","Data":"a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918"}
Oct 31 19:47:49.504295 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.504209    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" event={"ID":"6312c0d6-8021-4810-a98c-04aab3ad1e8e","Type":"ContainerStarted","Data":"1653072b01fcad9f46fa89fae3f75c8e30238eab6ef8ef3e3310f58619c78c3b"}
Oct 31 19:47:49.505702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.505675    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" event={"ID":"2a278f1d-df5d-40be-9b77-9292e1c8bc55","Type":"ContainerStarted","Data":"0e06e5920eeb4a0b3ef877bf05d417094757d9efbcdb756e2360319028079367"}
Oct 31 19:47:49.540029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.539998    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:47:49.573076 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:47:49.573045    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222bdfa6_5c87_47d5_9091_c60527060166.slice/crio-d7966698708ab0ba5adc49a99393b16d9862cfbe86e3de4783d9d0183c693140 WatchSource:0}: Error finding container d7966698708ab0ba5adc49a99393b16d9862cfbe86e3de4783d9d0183c693140: Status 404 returned error can't find the container with id d7966698708ab0ba5adc49a99393b16d9862cfbe86e3de4783d9d0183c693140
Oct 31 19:47:49.579692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.576415    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:49.579692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.576502    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:49.579692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.576546    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:49.579692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.576767    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"]
Oct 31 19:47:49.629124 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.629088    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a8f083-07f6-4101-bbc9-76b9797e5593" path="/var/lib/kubelet/pods/09a8f083-07f6-4101-bbc9-76b9797e5593/volumes"
Oct 31 19:47:49.747687 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.747331    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:49.802680 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.802160    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec52aa5f-4450-4549-828a-ac1b3ae29b39-kube-api-access-jtp82\" (UniqueName: \"kubernetes.io/projected/ec52aa5f-4450-4549-828a-ac1b3ae29b39-kube-api-access-jtp82\") pod \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") "
Oct 31 19:47:49.802680 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.802366    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-openshift-ca\") pod \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") "
Oct 31 19:47:49.802680 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.802548    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-xks-ca\") pod \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\" (UID: \"ec52aa5f-4450-4549-828a-ac1b3ae29b39\") "
Oct 31 19:47:49.835726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.823928    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-openshift-ca" pod "ec52aa5f-4450-4549-828a-ac1b3ae29b39" (UID: "ec52aa5f-4450-4549-828a-ac1b3ae29b39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:49.836364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.836327    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-xks-ca" pod "ec52aa5f-4450-4549-828a-ac1b3ae29b39" (UID: "ec52aa5f-4450-4549-828a-ac1b3ae29b39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:49.838255 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.838218    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec52aa5f-4450-4549-828a-ac1b3ae29b39-kube-api-access-jtp82" pod "ec52aa5f-4450-4549-828a-ac1b3ae29b39" (UID: "ec52aa5f-4450-4549-828a-ac1b3ae29b39"). InnerVolumeSpecName "kube-api-access-jtp82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:49.883837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.882440    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podStartSLOduration=1.882417155 podStartE2EDuration="1.882417155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:49.669169418 +0000 UTC m=+1888.198703496" watchObservedRunningTime="2026-10-31 19:47:49.882417155 +0000 UTC m=+1888.411951209"
Oct 31 19:47:49.892333 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.892300    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"]
Oct 31 19:47:49.893068 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.893045    5006 cpu_manager.go:414] "containerMap: removing container" podUID="ec52aa5f-4450-4549-828a-ac1b3ae29b39" containerName="revoke-keys"
Oct 31 19:47:49.893068 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.893067    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec52aa5f-4450-4549-828a-ac1b3ae29b39" containerName="revoke-keys"
Oct 31 19:47:49.893300 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.893285    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec52aa5f-4450-4549-828a-ac1b3ae29b39" containerName="revoke-keys"
Oct 31 19:47:49.909125 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.907857    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:49.909125 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.907896    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec52aa5f-4450-4549-828a-ac1b3ae29b39-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:49.909125 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.907916    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtp82\" (UniqueName: \"kubernetes.io/projected/ec52aa5f-4450-4549-828a-ac1b3ae29b39-kube-api-access-jtp82\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:49.935968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.935027    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:49.944865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:49.944741    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"]
Oct 31 19:47:50.027883 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.027727    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blqg\" (UniqueName: \"kubernetes.io/projected/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-kube-api-access-2blqg\") pod \"maas-api-revoke-sub-3c3baecb-qrp59\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:50.029053 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.028799    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c3baecb-qrp59\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:50.029979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.029868    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c3baecb-qrp59\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:50.032675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.032045    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"]
Oct 31 19:47:50.039095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.039051    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.087518 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.086527    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"]
Oct 31 19:47:50.140983 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.140938    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.141385 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.141355    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.144524 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.141661    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c3baecb-qrp59\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:50.147756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.147593    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c3baecb-qrp59\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:50.149397 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.149331    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsz6\" (UniqueName: \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-kube-api-access-ffsz6\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.149716 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.149696    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f72d787-e063-405e-b385-636791c04833-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.149972 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.149955    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blqg\" (UniqueName: \"kubernetes.io/projected/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-kube-api-access-2blqg\") pod \"maas-api-revoke-sub-3c3baecb-qrp59\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:50.150702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.150620    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f72d787-e063-405e-b385-636791c04833-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.150965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.150946    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.151123 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.151107    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.151267 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.151252    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.151429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.151394    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c3baecb-qrp59\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:50.151717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.151698    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.159782 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.159748    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c3baecb-qrp59\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:50.231240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.231201    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blqg\" (UniqueName: \"kubernetes.io/projected/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-kube-api-access-2blqg\") pod \"maas-api-revoke-sub-3c3baecb-qrp59\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:50.253757 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.253720    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsz6\" (UniqueName: \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-kube-api-access-ffsz6\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.253976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.253772    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f72d787-e063-405e-b385-636791c04833-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.253976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.253864    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f72d787-e063-405e-b385-636791c04833-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.253976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.253933    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.253976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.253966    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.254205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.254000    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.254205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.254102    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.254205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.254134    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.254205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.254196    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.254574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.254332    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.254638 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.254620    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.254750 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.254642    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.254750 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.254724    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f72d787-e063-405e-b385-636791c04833-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.254824 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.254760    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.256876 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.256852    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.256979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.256939    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f72d787-e063-405e-b385-636791c04833-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.262957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.262936    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:50.329552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.329503    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.329795 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.329770    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsz6\" (UniqueName: \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-kube-api-access-ffsz6\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.355891 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.355865    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:50.537498 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.537399    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r" event={"ID":"ec52aa5f-4450-4549-828a-ac1b3ae29b39","Type":"ContainerDied","Data":"b1972fb08c342ebcf3766b78e08e88e8ae5f221e03027cd14535e14443a82ccb"}
Oct 31 19:47:50.538998 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.538167    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1972fb08c342ebcf3766b78e08e88e8ae5f221e03027cd14535e14443a82ccb"
Oct 31 19:47:50.538998 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.538124    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"
Oct 31 19:47:50.541617 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.541101    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:47:50.541617 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.541160    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:47:50.548410 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.546423    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" event={"ID":"222bdfa6-5c87-47d5-9091-c60527060166","Type":"ContainerStarted","Data":"de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7"}
Oct 31 19:47:50.548410 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.546489    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" event={"ID":"222bdfa6-5c87-47d5-9091-c60527060166","Type":"ContainerStarted","Data":"d7966698708ab0ba5adc49a99393b16d9862cfbe86e3de4783d9d0183c693140"}
Oct 31 19:47:50.551597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.551508    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" event={"ID":"2a278f1d-df5d-40be-9b77-9292e1c8bc55","Type":"ContainerStarted","Data":"945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31"}
Oct 31 19:47:50.764321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.764270    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podStartSLOduration=2.76425447 podStartE2EDuration="2.76425447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:50.690617256 +0000 UTC m=+1889.220151305" watchObservedRunningTime="2026-10-31 19:47:50.76425447 +0000 UTC m=+1889.293788523"
Oct 31 19:47:50.769498 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.769474    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"]
Oct 31 19:47:50.866424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.866343    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"]
Oct 31 19:47:50.868214 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.868175    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:50.868425 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.868404    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:50.868557 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.868540    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:50.874162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:50.874113    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podStartSLOduration=2.874096803 podStartE2EDuration="2.874096803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:50.85569197 +0000 UTC m=+1889.385226023" watchObservedRunningTime="2026-10-31 19:47:50.874096803 +0000 UTC m=+1889.403630858"
Oct 31 19:47:51.095033 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.095006    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:47:51.132679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.119068    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:47:51.132679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.119127    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:47:51.132679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.121036    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:47:51.132679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.121150    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:47:51.132679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.121189    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:47:51.488738 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.463269    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:51.621681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.586890    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:47:51.621681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.586955    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:47:51.848291 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.818669    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59" event={"ID":"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9","Type":"ContainerStarted","Data":"1f938571867107413e6b557a63e3b6bd03123aa4ee608523377f826c639fd3eb"}
Oct 31 19:47:51.848291 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.818747    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59" event={"ID":"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9","Type":"ContainerStarted","Data":"98379ba38304c348e4a98edd26fe0083e2ed3d827c955324919dbe2901cab78c"}
Oct 31 19:47:51.970268 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.970233    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" event={"ID":"2f72d787-e063-405e-b385-636791c04833","Type":"ContainerStarted","Data":"d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6"}
Oct 31 19:47:51.993808 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.993756    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" event={"ID":"2f72d787-e063-405e-b385-636791c04833","Type":"ContainerStarted","Data":"3cdb7cec925b97a04572798af5f3650c933c99e9061a90b885b7495fce8fafc6"}
Oct 31 19:47:51.994064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:51.976430    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59" podStartSLOduration=2.976411912 podStartE2EDuration="2.976411912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:51.974862159 +0000 UTC m=+1890.504396240" watchObservedRunningTime="2026-10-31 19:47:51.976411912 +0000 UTC m=+1890.505945965"
Oct 31 19:47:52.120356 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.120214    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podStartSLOduration=3.120191007 podStartE2EDuration="3.120191007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:52.092496189 +0000 UTC m=+1890.622030246" watchObservedRunningTime="2026-10-31 19:47:52.120191007 +0000 UTC m=+1890.649725062"
Oct 31 19:47:52.151126 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.126806    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:47:52.151424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.151388    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:47:52.175934 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.154800    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:47:52.175934 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.154855    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:47:52.366083 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.364056    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:47:52.366274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.366206    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:47:52.366274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.366258    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:47:52.550888 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.550794    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:47:52.551143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.551115    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:47:52.661147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.661099    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:47:52.765678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.761482    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"]
Oct 31 19:47:52.787686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.775903    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:52.940676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.882344    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc06507-zxkt7\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:52.940676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.882428    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc06507-zxkt7\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:52.940676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:52.882509    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfsjm\" (UniqueName: \"kubernetes.io/projected/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-kube-api-access-tfsjm\") pod \"maas-api-revoke-sub-2cc06507-zxkt7\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:53.083793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.083731    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:47:53.113931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.112487    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:47:53.113931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.112548    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:47:53.126692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.122112    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:47:53.126692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.122173    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:47:53.126692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.122783    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc06507-zxkt7\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:53.126692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.122858    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc06507-zxkt7\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:53.126692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.122967    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfsjm\" (UniqueName: \"kubernetes.io/projected/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-kube-api-access-tfsjm\") pod \"maas-api-revoke-sub-2cc06507-zxkt7\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:53.126692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.124881    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cc06507-zxkt7\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:53.126692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.126253    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:53.175674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.154162    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cc06507-zxkt7\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:53.175674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.158159    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:53.179681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.176303    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"]
Oct 31 19:47:53.199878 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.199753    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:53.228677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.201530    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:53.229788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.229739    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:53.257487 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.257451    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfsjm\" (UniqueName: \"kubernetes.io/projected/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-kube-api-access-tfsjm\") pod \"maas-api-revoke-sub-2cc06507-zxkt7\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:53.305692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.304759    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:53.338732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.321479    5006 generic.go:356] "Generic (PLEG): container finished" podID="a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9" containerID="1f938571867107413e6b557a63e3b6bd03123aa4ee608523377f826c639fd3eb" exitCode=0
Oct 31 19:47:53.338732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.323268    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59" event={"ID":"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9","Type":"ContainerDied","Data":"1f938571867107413e6b557a63e3b6bd03123aa4ee608523377f826c639fd3eb"}
Oct 31 19:47:53.404691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.400984    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:47:53.404691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.401093    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:47:53.416771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.413963    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:53.439703 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.432792    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:53.482544 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.482437    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:47:53.524708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.524666    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:53.565731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.553779    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:47:53.565731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.554207    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:47:53.565731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.554468    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:53.656677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.653378    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": context deadline exceeded"
Oct 31 19:47:53.670672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:53.664030    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:54.070737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.070171    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:54.070737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.070587    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Oct 31 19:47:54.138672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.117128    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:47:54.138672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.117196    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:47:54.138672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.119800    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:47:54.138672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.119848    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:47:54.138672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.119915    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:54.140948 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.139411    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:54.140948 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.140301    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:54.142438 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.142415    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:47:54.152072 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.142996    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:47:54.205729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.205697    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:47:54.206708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.180397    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:54.208162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.208134    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:47:54.234890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.234864    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"]
Oct 31 19:47:54.236088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.234785    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:47:54.236584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.234741    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:54.237520 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.236759    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:47:54.238180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.238164    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:47:54.315674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.312750    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:54.315674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.313092    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:47:54.577694 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.575665    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:47:54.577694 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.576100    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:47:54.581689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.580560    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:47:54.581689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.580933    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded"
Oct 31 19:47:54.581689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.580994    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:47:54.581689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.581246    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"]
Oct 31 19:47:54.581689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.581323    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:47:54.581689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.581368    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:47:54.690928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.687862    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:54.693279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.692266    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:47:54.761798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:54.755485    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:47:55.052537 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.052446    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955eb881-bt54f"]
Oct 31 19:47:55.085746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.085698    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 31 19:47:55.087590 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.087558    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.128833    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.128919    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.155800    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.155867    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.157608    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.157682    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.157761    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.158995    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.160187    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.162508    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:47:55.181339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.178820    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:47:55.225628 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.225585    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:47:55.225931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.225898    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:47:55.251745 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.246822    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:55.306574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.263541    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:55.318684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.308790    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:55.560775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.501911    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 31 19:47:55.560775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.535307    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:47:55.560775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.535366    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:47:55.562245 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.562087    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:47:55.562245 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.562169    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:47:55.581029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.580960    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:55.582575 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.582495    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:55.604544 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.595802    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:47:55.747995 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.739998    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:55.868977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.807882    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:47:56.029625 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:55.974298    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:56.121742 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.110437    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:56.148239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.148198    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:47:56.151781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.151730    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:47:56.166296 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.165681    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:47:56.167768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.165616    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:47:56.167768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.165713    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:47:56.190106 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.190069    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:47:56.190388 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.190342    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:47:56.311680 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.286703    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:47:56.340528 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.340489    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded"
Oct 31 19:47:56.340827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.340802    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 31 19:47:56.341314 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.341281    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:56.497232 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.458079    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:47:56.531232 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.531197    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:47:56.531543 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.531104    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:56.531972 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.487530    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1406693d-432d-4f84-9d81-d0bdc062f65c" path="/var/lib/kubelet/pods/1406693d-432d-4f84-9d81-d0bdc062f65c/volumes"
Oct 31 19:47:56.654050 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:47:56.654003    5006 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.061s"
Oct 31 19:47:56.654301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.654282    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ddd4236-4f686"]
Oct 31 19:47:56.673952 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.617147    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:56.674524 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.617187    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:47:56.675594 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.675563    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:56.676221 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.617067    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:56.706048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.705995    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:47:56.720990 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.720945    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:47:56.749886 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.749767    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:56.866081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.808162    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:47:56.866081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:56.822960    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:57.110735 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.110675    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:57.286165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.197747    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:57.294690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.197788    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:47:57.317583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.197808    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:47:57.318931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.249988    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:47:57.318931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.318316    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:47:57.318931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.288021    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:47:57.318931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.318396    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:47:57.344676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.319922    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:47:57.344969 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.344923    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:57.348615 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.346320    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:57.405958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.402774    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:47:57.405958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.402838    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:47:57.405958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.405033    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7" event={"ID":"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939","Type":"ContainerStarted","Data":"a518a540446adf19a62d92eaec079c03bddd8d1ff6ae023e6497d1c07e9434ac"}
Oct 31 19:47:57.502796 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.502746    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:57.536804 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.536749    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 31 19:47:57.641679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.636329    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:47:57.641679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.636389    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:47:57.690741 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.684831    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:57.730272 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.730043    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:47:57.848684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.815759    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:47:57.864620 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.837275    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.040736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:57.927258    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9558d2b-5fbd-45bb-a667-b28220cb402d" path="/var/lib/kubelet/pods/a9558d2b-5fbd-45bb-a667-b28220cb402d/volumes"
Oct 31 19:47:58.156855 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.156813    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:47:58.203682 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.174425    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:47:58.204173 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.164794    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"]
Oct 31 19:47:58.244979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.228357    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:47:58.245160 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.245132    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:47:58.402292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.317373    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.402292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.323749    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.402292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.324209    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.402292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.343790    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.402292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.357265    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:47:58.402292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.358691    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 31 19:47:58.402292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.361880    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:47:58.402292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.361932    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:47:58.410691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.407718    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:47:58.410691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.407789    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.508539 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.508484    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.567851 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.542723    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:47:58.567851 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.542789    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:47:58.567851 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.542868    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.651913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.651794    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4001af4c-pmtrz"]
Oct 31 19:47:58.653525 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.653485    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59" event={"ID":"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9","Type":"ContainerDied","Data":"98379ba38304c348e4a98edd26fe0083e2ed3d827c955324919dbe2901cab78c"}
Oct 31 19:47:58.653820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.653796    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98379ba38304c348e4a98edd26fe0083e2ed3d827c955324919dbe2901cab78c"
Oct 31 19:47:58.659943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.659899    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:47:58.680137 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.680096    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-kube-api-access-2blqg\" (UniqueName: \"kubernetes.io/projected/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-kube-api-access-2blqg\") pod \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") "
Oct 31 19:47:58.680975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.680946    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-openshift-ca\") pod \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") "
Oct 31 19:47:58.681374 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.681354    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-xks-ca\") pod \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\" (UID: \"a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9\") "
Oct 31 19:47:58.682029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.681992    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-openshift-ca" pod "a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9" (UID: "a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:58.690442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.690400    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 31 19:47:58.735593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.735549    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:58.799890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.799846    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-xks-ca" pod "a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9" (UID: "a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:58.842732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.824342    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.853363 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.852462    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:58.867780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.867734    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.893214 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.893170    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:47:58.893509 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.893480    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:58.906025 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:58.905920    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:59.090787 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.090730    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-kube-api-access-2blqg" pod "a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9" (UID: "a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9"). InnerVolumeSpecName "kube-api-access-2blqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:59.144194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.144155    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:47:59.144464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.144431    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:47:59.216801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.214371    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2blqg\" (UniqueName: \"kubernetes.io/projected/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9-kube-api-access-2blqg\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:59.232710 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.222095    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"]
Oct 31 19:47:59.233089 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.233051    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:47:59.233351 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.233313    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:47:59.304417 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.304380    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:47:59.366920 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.366852    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:59.367792 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.367744    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:47:59.368412 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.368373    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:47:59.382696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.382005    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:47:59.390262 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.390207    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 31 19:47:59.415154 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.398804    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:47:59.415706 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.398871    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:47:59.458846 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.458794    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:59.461643 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.399747    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:47:59.759565 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.759522    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:47:59.759942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.518008    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:59.760405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.625851    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:59.766641 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.666844    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:59.827098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.461474    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:59.852792 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.718624    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:47:59.853137 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.853100    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:47:59.853266 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.826530    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:59.896119 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.896077    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:47:59.970707 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:47:59.963577    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 31 19:48:00.194741 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.182323    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.194741 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.183679    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:48:00.194741 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.183749    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:48:00.214607 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.214559    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:48:00.214934 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.214880    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:48:00.276677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.247690    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda96c2d-0405-4840-9836-6357f68f9b68" path="/var/lib/kubelet/pods/cda96c2d-0405-4840-9836-6357f68f9b68/volumes"
Oct 31 19:48:00.373418 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.373372    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.387679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.374972    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.387679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.376698    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"
Oct 31 19:48:00.387679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.377307    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7" event={"ID":"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939","Type":"ContainerStarted","Data":"3f7e8ac27f27a8528adb4b11b0a306ba1029fe97ad5c3b14af4f923948ed5285"}
Oct 31 19:48:00.417976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.413851    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.417976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.415916    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.417976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.416253    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 31 19:48:00.418384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.418348    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.423668 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.423605    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:48:00.423929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.423895    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:48:00.608901 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.608856    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:48:00.609213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.609178    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:48:00.737148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.737072    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.769447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.769385    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.797678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.772837    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:48:00.867740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.802536    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.867740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.834228    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 31 19:48:00.867740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.854416    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.915677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.904406    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:00.950677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.934252    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:00.999685 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:00.993733    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:01.135711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.112834    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:48:01.206686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.141803    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:48:01.206686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.152933    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:48:01.206686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.152992    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:48:01.247677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.243747    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:01.385890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.385780    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:48:01.411129 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.411089    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:48:01.419678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.413166    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:48:01.419963 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.392706    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 31 19:48:01.439124 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.417793    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:01.439124 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.417827    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:48:01.439124 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.417848    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:48:01.439124 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.418763    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:48:01.477527 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.477477    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:01.499845 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:48:01.499617    5006 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6803fa_8a7e_44b1_b2d1_ed9d6179f939.slice/crio-conmon-3f7e8ac27f27a8528adb4b11b0a306ba1029fe97ad5c3b14af4f923948ed5285.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6803fa_8a7e_44b1_b2d1_ed9d6179f939.slice/crio-3f7e8ac27f27a8528adb4b11b0a306ba1029fe97ad5c3b14af4f923948ed5285.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54fdb88_3a3a_43fb_9579_ccc39eb5b6a9.slice/crio-98379ba38304c348e4a98edd26fe0083e2ed3d827c955324919dbe2901cab78c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54fdb88_3a3a_43fb_9579_ccc39eb5b6a9.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 19:48:01.555677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.551797    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:48:01.563677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.559736    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:48:01.567677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.561950    5006 generic.go:356] "Generic (PLEG): container finished" podID="0a6803fa-8a7e-44b1-b2d1-ed9d6179f939" containerID="3f7e8ac27f27a8528adb4b11b0a306ba1029fe97ad5c3b14af4f923948ed5285" exitCode=0
Oct 31 19:48:01.593031 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.561982    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7" event={"ID":"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939","Type":"ContainerDied","Data":"3f7e8ac27f27a8528adb4b11b0a306ba1029fe97ad5c3b14af4f923948ed5285"}
Oct 31 19:48:01.786187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.754081    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:01.807678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.770762    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:48:01.858677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.846759    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:48:01.861767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.859043    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:48:01.867808 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.867749    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:01.952674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.934855    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:01.971529 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:01.971489    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:48:02.030681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.014799    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:02.192426 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.192368    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:48:02.445737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.439249    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:48:02.445737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.439062    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:02.445737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.203126    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:48:02.445737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.439717    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:48:02.450094 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.450047    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 31 19:48:02.472278 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.472227    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:48:02.538677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.486970    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:48:02.538677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.487041    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:48:02.538677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.487089    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:02.538677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.487733    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:02.569282 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.563794    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:48:02.624522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.624480    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:48:02.643727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.638778    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 19:48:02.656686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.650530    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 19:48:02.792001 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.791863    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:02.797748 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.797498    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:02.894706 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.867046    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:48:02.894706 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.880585    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:02.990863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:02.978004    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:03.075768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.041119    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:03.468125 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.443462    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:03.468125 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.462608    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 31 19:48:03.492195 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.492157    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:48:03.492483 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.492466    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:48:03.555257 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.555225    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:48:03.560749 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.511257    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:48:03.565252 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.539492    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:48:03.657624 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.626289    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:48:03.672364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.672139    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:03.697718 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.694768    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:48:03.745986 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.745932    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:48:03.746323 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.746291    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:48:03.750120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.750086    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-xks-ca\") pod \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") "
Oct 31 19:48:03.780675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.759838    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-openshift-ca\") pod \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") "
Oct 31 19:48:03.782167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.782133    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-kube-api-access-tfsjm\" (UniqueName: \"kubernetes.io/projected/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-kube-api-access-tfsjm\") pod \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\" (UID: \"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939\") "
Oct 31 19:48:03.789701 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.786200    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:48:03.793279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.793235    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:03.929121 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.765234    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-openshift-ca" pod "0a6803fa-8a7e-44b1-b2d1-ed9d6179f939" (UID: "0a6803fa-8a7e-44b1-b2d1-ed9d6179f939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:03.929121 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.794845    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-kube-api-access-tfsjm" pod "0a6803fa-8a7e-44b1-b2d1-ed9d6179f939" (UID: "0a6803fa-8a7e-44b1-b2d1-ed9d6179f939"). InnerVolumeSpecName "kube-api-access-tfsjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:03.929121 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.787916    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 31 19:48:03.937544 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.937480    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:48:03.937823 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.937788    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:48:03.939319 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.939287    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:48:03.978313 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.978199    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:03.988172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:03.988139    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"
Oct 31 19:48:04.137695 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.123042    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:48:04.137695 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.123107    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 19:48:04.137695 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.123409    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-xks-ca" pod "0a6803fa-8a7e-44b1-b2d1-ed9d6179f939" (UID: "0a6803fa-8a7e-44b1-b2d1-ed9d6179f939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:04.179393 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.179353    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:48:04.237146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.202296    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"]
Oct 31 19:48:04.237781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.237751    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:48:04.237945 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.237932    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:48:04.238114 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.238100    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad5b2cc-7k46f"]
Oct 31 19:48:04.238236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.238214    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7" event={"ID":"0a6803fa-8a7e-44b1-b2d1-ed9d6179f939","Type":"ContainerDied","Data":"a518a540446adf19a62d92eaec079c03bddd8d1ff6ae023e6497d1c07e9434ac"}
Oct 31 19:48:04.238378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.238361    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a518a540446adf19a62d92eaec079c03bddd8d1ff6ae023e6497d1c07e9434ac"
Oct 31 19:48:04.238480 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.238469    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:48:04.238704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.238678    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 19:48:04.265675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.261776    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:04.290879 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.290816    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:04.294991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.294101    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:48:04.353672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.349939    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:48:04.357854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.357810    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:48:04.359446 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.359418    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:48:04.363877 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.361997    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:48:04.364123 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.364098    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:48:04.364226 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.363139    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:48:04.364540 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.362153    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:04.381726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.381693    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfsjm\" (UniqueName: \"kubernetes.io/projected/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-kube-api-access-tfsjm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:04.409671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.407260    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:48:04.409671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.407958    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:04.409671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.407983    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:04.423406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.423376    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:48:04.437227 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.437198    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:48:04.469581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.469553    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:48:04.470749 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.470726    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"]
Oct 31 19:48:04.504033 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.503865    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d2c7f2-c5krx"]
Oct 31 19:48:04.570971 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.570942    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:48:04.572886 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.572462    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:48:04.595233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.595137    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7" podStartSLOduration=12.595116314 podStartE2EDuration="12.595116314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:00.567196746 +0000 UTC m=+1899.096730800" watchObservedRunningTime="2026-10-31 19:48:04.595116314 +0000 UTC m=+1903.124650367"
Oct 31 19:48:04.618125 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.618091    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 19:48:04.683085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.682445    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"]
Oct 31 19:48:04.689238 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:04.689102    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4af8d2-jlck7"]
Oct 31 19:48:05.369419 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:05.369190    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:48:05.396690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:05.370165    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 19:48:05.396690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:05.370222    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 19:48:05.544737 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:48:05.543858    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:48:05.558760 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:05.558512    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:48:05.565059 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:05.564965    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:48:05.605109 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:05.605068    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f7e061-d824-460b-8030-fb6842528190" path="/var/lib/kubelet/pods/27f7e061-d824-460b-8030-fb6842528190/volumes"
Oct 31 19:48:05.606723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:05.606694    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7628c4-4ae8-4294-8d26-a6799f43c14b" path="/var/lib/kubelet/pods/4d7628c4-4ae8-4294-8d26-a6799f43c14b/volumes"
Oct 31 19:48:05.607674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:05.607626    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86bcc1a5-0be2-46eb-95bc-79425cb3a39e" path="/var/lib/kubelet/pods/86bcc1a5-0be2-46eb-95bc-79425cb3a39e/volumes"
Oct 31 19:48:06.164874 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.162955    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"]
Oct 31 19:48:06.220628 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.220563    5006 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:48:06.234269 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.234214    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:06.321505 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.321465    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:06.478985 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.478878    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359fcc72-wz5bf"]
Oct 31 19:48:06.944702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.941427    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"]
Oct 31 19:48:06.944702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.942585    5006 cpu_manager.go:414] "containerMap: removing container" podUID="a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9" containerName="revoke-keys"
Oct 31 19:48:06.944702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.942613    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9" containerName="revoke-keys"
Oct 31 19:48:06.944702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.942729    5006 cpu_manager.go:414] "containerMap: removing container" podUID="0a6803fa-8a7e-44b1-b2d1-ed9d6179f939" containerName="revoke-keys"
Oct 31 19:48:06.944702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.942741    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a6803fa-8a7e-44b1-b2d1-ed9d6179f939" containerName="revoke-keys"
Oct 31 19:48:06.944702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.942986    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a6803fa-8a7e-44b1-b2d1-ed9d6179f939" containerName="revoke-keys"
Oct 31 19:48:06.944702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.943007    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9" containerName="revoke-keys"
Oct 31 19:48:06.951677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:06.949415    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.011754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.002536    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:48:07.011754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.005666    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:48:07.011754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.006389    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:48:07.011754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.006473    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:48:07.011754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.006491    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:48:07.011754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.007508    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"]
Oct 31 19:48:07.019541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.014513    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:48:07.057672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.057410    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e348152-v4xpg\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.057672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.057513    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e348152-v4xpg\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.057672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.057559    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w52fx\" (UniqueName: \"kubernetes.io/projected/99e14978-4d67-42e1-93a6-268a6deedd69-kube-api-access-w52fx\") pod \"maas-api-revoke-sub-6e348152-v4xpg\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.167923 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.167877    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e348152-v4xpg\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.168114 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.167977    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e348152-v4xpg\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.168114 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.168013    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w52fx\" (UniqueName: \"kubernetes.io/projected/99e14978-4d67-42e1-93a6-268a6deedd69-kube-api-access-w52fx\") pod \"maas-api-revoke-sub-6e348152-v4xpg\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.169205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.169172    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e348152-v4xpg\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.183671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.176524    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e348152-v4xpg\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.248720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.248162    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w52fx\" (UniqueName: \"kubernetes.io/projected/99e14978-4d67-42e1-93a6-268a6deedd69-kube-api-access-w52fx\") pod \"maas-api-revoke-sub-6e348152-v4xpg\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.284734 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.284696    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:07.407827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.407534    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:48:07.407827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.407610    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:07.430674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.425850    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"]
Oct 31 19:48:07.461391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.461359    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.485377 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.485315    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"]
Oct 31 19:48:07.557762 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.557641    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"]
Oct 31 19:48:07.563741 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.563705    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:07.578960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.578866    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777f4f51-efa9-4be1-a86f-626692778040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa4c76c9-xcb24\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.578960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.578921    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777f4f51-efa9-4be1-a86f-626692778040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa4c76c9-xcb24\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.579325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.579045    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtm9h\" (UniqueName: \"kubernetes.io/projected/777f4f51-efa9-4be1-a86f-626692778040-kube-api-access-xtm9h\") pod \"maas-api-revoke-sub-aa4c76c9-xcb24\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.608726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.608120    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd15eca-c23e-4dee-a552-9295213ffa66" path="/var/lib/kubelet/pods/6fd15eca-c23e-4dee-a552-9295213ffa66/volumes"
Oct 31 19:48:07.613702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.613662    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"]
Oct 31 19:48:07.680605 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.680569    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777f4f51-efa9-4be1-a86f-626692778040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa4c76c9-xcb24\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.680834 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.680610    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777f4f51-efa9-4be1-a86f-626692778040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa4c76c9-xcb24\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.680834 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.680665    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtm9h\" (UniqueName: \"kubernetes.io/projected/777f4f51-efa9-4be1-a86f-626692778040-kube-api-access-xtm9h\") pod \"maas-api-revoke-sub-aa4c76c9-xcb24\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.680834 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.680710    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e29362c-pkrvd\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:07.680834 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.680820    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbn5\" (UniqueName: \"kubernetes.io/projected/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-kube-api-access-gkbn5\") pod \"maas-api-revoke-sub-0e29362c-pkrvd\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:07.681132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.681111    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e29362c-pkrvd\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:07.681409 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.681386    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777f4f51-efa9-4be1-a86f-626692778040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa4c76c9-xcb24\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.683185 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.683167    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777f4f51-efa9-4be1-a86f-626692778040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa4c76c9-xcb24\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.738233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.738194    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtm9h\" (UniqueName: \"kubernetes.io/projected/777f4f51-efa9-4be1-a86f-626692778040-kube-api-access-xtm9h\") pod \"maas-api-revoke-sub-aa4c76c9-xcb24\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.765597 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:07.765557    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e14978_4d67_42e1_93a6_268a6deedd69.slice/crio-fff6f4a91e95529fe3a2bdbe794de8ce340ade92124f2cdfdf664130d4135c12 WatchSource:0}: Error finding container fff6f4a91e95529fe3a2bdbe794de8ce340ade92124f2cdfdf664130d4135c12: Status 404 returned error can't find the container with id fff6f4a91e95529fe3a2bdbe794de8ce340ade92124f2cdfdf664130d4135c12
Oct 31 19:48:07.769068 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.769044    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"]
Oct 31 19:48:07.786413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.782508    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e29362c-pkrvd\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:07.786413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.782636    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbn5\" (UniqueName: \"kubernetes.io/projected/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-kube-api-access-gkbn5\") pod \"maas-api-revoke-sub-0e29362c-pkrvd\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:07.786413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.784263    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e29362c-pkrvd\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:07.786413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.785239    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e29362c-pkrvd\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:07.787812 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.787785    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e29362c-pkrvd\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:07.801038 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.800998    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:07.841894 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.841813    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbn5\" (UniqueName: \"kubernetes.io/projected/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-kube-api-access-gkbn5\") pod \"maas-api-revoke-sub-0e29362c-pkrvd\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:07.884188 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:07.883878    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:08.186885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.186851    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:48:08.189239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.189155    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:48:08.210888 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.210854    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"]
Oct 31 19:48:08.371873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.371835    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg" event={"ID":"99e14978-4d67-42e1-93a6-268a6deedd69","Type":"ContainerStarted","Data":"90bf54e3979b8cbf6d3e935b4c339201dbabbe00dd564a9daff06dfd1415864c"}
Oct 31 19:48:08.372062 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.371895    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg" event={"ID":"99e14978-4d67-42e1-93a6-268a6deedd69","Type":"ContainerStarted","Data":"fff6f4a91e95529fe3a2bdbe794de8ce340ade92124f2cdfdf664130d4135c12"}
Oct 31 19:48:08.373408 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.373379    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24" event={"ID":"777f4f51-efa9-4be1-a86f-626692778040","Type":"ContainerStarted","Data":"0bec4f67a61751ee1d7039ecd74edda8a2d1c885720cb2fb2d55bc0ac42be39d"}
Oct 31 19:48:08.413250 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.413204    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"]
Oct 31 19:48:08.545148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.545092    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg" podStartSLOduration=2.545076671 podStartE2EDuration="2.545076671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:08.542971837 +0000 UTC m=+1907.072505890" watchObservedRunningTime="2026-10-31 19:48:08.545076671 +0000 UTC m=+1907.074610777"
Oct 31 19:48:08.608262 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.607528    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"]
Oct 31 19:48:08.612636 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.612614    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:08.672563 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.672470    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"]
Oct 31 19:48:08.710094 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.710053    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4kg\" (UniqueName: \"kubernetes.io/projected/a5f9d736-7844-4ddc-861e-007871cfa749-kube-api-access-8n4kg\") pod \"maas-api-revoke-sub-e2adf7fb-lmx77\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:08.710269 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.710225    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2adf7fb-lmx77\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:08.710449 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.710420    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2adf7fb-lmx77\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:08.812387 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.812357    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2adf7fb-lmx77\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:08.812523 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.812493    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4kg\" (UniqueName: \"kubernetes.io/projected/a5f9d736-7844-4ddc-861e-007871cfa749-kube-api-access-8n4kg\") pod \"maas-api-revoke-sub-e2adf7fb-lmx77\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:08.812594 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.812576    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2adf7fb-lmx77\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:08.813140 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.813114    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2adf7fb-lmx77\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:08.814982 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.814962    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2adf7fb-lmx77\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:08.870113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.870078    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4kg\" (UniqueName: \"kubernetes.io/projected/a5f9d736-7844-4ddc-861e-007871cfa749-kube-api-access-8n4kg\") pod \"maas-api-revoke-sub-e2adf7fb-lmx77\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:08.928669 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:08.928552    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:09.329291 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:09.329253    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f9d736_7844_4ddc_861e_007871cfa749.slice/crio-3786bfd064295da99b82ffedcb05ffb06ee6c1b88e52d5528e6542e97bcd0a20 WatchSource:0}: Error finding container 3786bfd064295da99b82ffedcb05ffb06ee6c1b88e52d5528e6542e97bcd0a20: Status 404 returned error can't find the container with id 3786bfd064295da99b82ffedcb05ffb06ee6c1b88e52d5528e6542e97bcd0a20
Oct 31 19:48:09.329442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:09.329322    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"]
Oct 31 19:48:09.389240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:09.389206    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77" event={"ID":"a5f9d736-7844-4ddc-861e-007871cfa749","Type":"ContainerStarted","Data":"3786bfd064295da99b82ffedcb05ffb06ee6c1b88e52d5528e6542e97bcd0a20"}
Oct 31 19:48:09.390981 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:09.390951    5006 generic.go:356] "Generic (PLEG): container finished" podID="99e14978-4d67-42e1-93a6-268a6deedd69" containerID="90bf54e3979b8cbf6d3e935b4c339201dbabbe00dd564a9daff06dfd1415864c" exitCode=0
Oct 31 19:48:09.391130 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:09.391024    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg" event={"ID":"99e14978-4d67-42e1-93a6-268a6deedd69","Type":"ContainerDied","Data":"90bf54e3979b8cbf6d3e935b4c339201dbabbe00dd564a9daff06dfd1415864c"}
Oct 31 19:48:09.392642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:09.392619    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd" event={"ID":"196e332b-96d9-4f71-98ee-3ca1da1d5eb5","Type":"ContainerStarted","Data":"8fb7b90af1cbe8ecdbfb6f923dbf4e5f33bbffd3c2a17f1592f797c8c937012c"}
Oct 31 19:48:09.392783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:09.392694    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd" event={"ID":"196e332b-96d9-4f71-98ee-3ca1da1d5eb5","Type":"ContainerStarted","Data":"41f04bb60079eaeb7f295e202532b6f1c748ccab75b6bbe731fef39e699e20e9"}
Oct 31 19:48:09.394273 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:09.394246    5006 generic.go:356] "Generic (PLEG): container finished" podID="777f4f51-efa9-4be1-a86f-626692778040" containerID="4ef9e962132b8a5ee4e776c2e5cf328fb21081a3d159240ff009e302897b7b61" exitCode=0
Oct 31 19:48:09.394352 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:09.394288    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24" event={"ID":"777f4f51-efa9-4be1-a86f-626692778040","Type":"ContainerDied","Data":"4ef9e962132b8a5ee4e776c2e5cf328fb21081a3d159240ff009e302897b7b61"}
Oct 31 19:48:09.752993 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:09.752888    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd" podStartSLOduration=2.752872243 podStartE2EDuration="2.752872243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:09.749250223 +0000 UTC m=+1908.278784277" watchObservedRunningTime="2026-10-31 19:48:09.752872243 +0000 UTC m=+1908.282406292"
Oct 31 19:48:10.404600 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.404563    5006 generic.go:356] "Generic (PLEG): container finished" podID="196e332b-96d9-4f71-98ee-3ca1da1d5eb5" containerID="8fb7b90af1cbe8ecdbfb6f923dbf4e5f33bbffd3c2a17f1592f797c8c937012c" exitCode=0
Oct 31 19:48:10.404835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.404665    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd" event={"ID":"196e332b-96d9-4f71-98ee-3ca1da1d5eb5","Type":"ContainerDied","Data":"8fb7b90af1cbe8ecdbfb6f923dbf4e5f33bbffd3c2a17f1592f797c8c937012c"}
Oct 31 19:48:10.406446 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.406422    5006 generic.go:356] "Generic (PLEG): container finished" podID="a5f9d736-7844-4ddc-861e-007871cfa749" containerID="be58b242282c3c424be09ba9bbef21094ff1267a8ea4f78ae4305e7d5f40eb00" exitCode=0
Oct 31 19:48:10.406558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.406460    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77" event={"ID":"a5f9d736-7844-4ddc-861e-007871cfa749","Type":"ContainerDied","Data":"be58b242282c3c424be09ba9bbef21094ff1267a8ea4f78ae4305e7d5f40eb00"}
Oct 31 19:48:10.666052 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.666023    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:10.670152 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.670114    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:10.751162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.751130    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777f4f51-efa9-4be1-a86f-626692778040-kube-api-access-xtm9h\" (UniqueName: \"kubernetes.io/projected/777f4f51-efa9-4be1-a86f-626692778040-kube-api-access-xtm9h\") pod \"777f4f51-efa9-4be1-a86f-626692778040\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") "
Oct 31 19:48:10.751341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.751201    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777f4f51-efa9-4be1-a86f-626692778040-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777f4f51-efa9-4be1-a86f-626692778040-maas-api-openshift-ca\") pod \"777f4f51-efa9-4be1-a86f-626692778040\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") "
Oct 31 19:48:10.751341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.751302    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-openshift-ca\") pod \"99e14978-4d67-42e1-93a6-268a6deedd69\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") "
Oct 31 19:48:10.751341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.751338    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-xks-ca\") pod \"99e14978-4d67-42e1-93a6-268a6deedd69\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") "
Oct 31 19:48:10.751491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.751367    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e14978-4d67-42e1-93a6-268a6deedd69-kube-api-access-w52fx\" (UniqueName: \"kubernetes.io/projected/99e14978-4d67-42e1-93a6-268a6deedd69-kube-api-access-w52fx\") pod \"99e14978-4d67-42e1-93a6-268a6deedd69\" (UID: \"99e14978-4d67-42e1-93a6-268a6deedd69\") "
Oct 31 19:48:10.751491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.751483    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777f4f51-efa9-4be1-a86f-626692778040-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777f4f51-efa9-4be1-a86f-626692778040-maas-api-xks-ca\") pod \"777f4f51-efa9-4be1-a86f-626692778040\" (UID: \"777f4f51-efa9-4be1-a86f-626692778040\") "
Oct 31 19:48:10.751685 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.751619    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777f4f51-efa9-4be1-a86f-626692778040-maas-api-openshift-ca" pod "777f4f51-efa9-4be1-a86f-626692778040" (UID: "777f4f51-efa9-4be1-a86f-626692778040"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:10.751775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.751750    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-openshift-ca" pod "99e14978-4d67-42e1-93a6-268a6deedd69" (UID: "99e14978-4d67-42e1-93a6-268a6deedd69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:10.752175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.752147    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777f4f51-efa9-4be1-a86f-626692778040-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:10.752286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.752179    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:10.753600 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.753571    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-xks-ca" pod "99e14978-4d67-42e1-93a6-268a6deedd69" (UID: "99e14978-4d67-42e1-93a6-268a6deedd69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:10.753600 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.753593    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777f4f51-efa9-4be1-a86f-626692778040-kube-api-access-xtm9h" pod "777f4f51-efa9-4be1-a86f-626692778040" (UID: "777f4f51-efa9-4be1-a86f-626692778040"). InnerVolumeSpecName "kube-api-access-xtm9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:10.753993 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.753619    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e14978-4d67-42e1-93a6-268a6deedd69-kube-api-access-w52fx" pod "99e14978-4d67-42e1-93a6-268a6deedd69" (UID: "99e14978-4d67-42e1-93a6-268a6deedd69"). InnerVolumeSpecName "kube-api-access-w52fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:10.753993 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.753721    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777f4f51-efa9-4be1-a86f-626692778040-maas-api-xks-ca" pod "777f4f51-efa9-4be1-a86f-626692778040" (UID: "777f4f51-efa9-4be1-a86f-626692778040"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:10.853730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.853631    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtm9h\" (UniqueName: \"kubernetes.io/projected/777f4f51-efa9-4be1-a86f-626692778040-kube-api-access-xtm9h\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:10.853730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.853678    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e14978-4d67-42e1-93a6-268a6deedd69-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:10.853730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.853688    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w52fx\" (UniqueName: \"kubernetes.io/projected/99e14978-4d67-42e1-93a6-268a6deedd69-kube-api-access-w52fx\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:10.853730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:10.853697    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777f4f51-efa9-4be1-a86f-626692778040-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.054508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.054471    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"]
Oct 31 19:48:11.059349 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.059313    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c6a600a-j69d2"]
Oct 31 19:48:11.191546 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.191455    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:11.417590 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.417561    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"
Oct 31 19:48:11.417590 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.417576    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg" event={"ID":"99e14978-4d67-42e1-93a6-268a6deedd69","Type":"ContainerDied","Data":"fff6f4a91e95529fe3a2bdbe794de8ce340ade92124f2cdfdf664130d4135c12"}
Oct 31 19:48:11.417867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.417714    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fff6f4a91e95529fe3a2bdbe794de8ce340ade92124f2cdfdf664130d4135c12"
Oct 31 19:48:11.419620 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.419595    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"
Oct 31 19:48:11.419806 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.419637    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24" event={"ID":"777f4f51-efa9-4be1-a86f-626692778040","Type":"ContainerDied","Data":"0bec4f67a61751ee1d7039ecd74edda8a2d1c885720cb2fb2d55bc0ac42be39d"}
Oct 31 19:48:11.419806 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.419733    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bec4f67a61751ee1d7039ecd74edda8a2d1c885720cb2fb2d55bc0ac42be39d"
Oct 31 19:48:11.605136 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.605103    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae6c4b7-98d7-4345-9c19-c42d18c90729" path="/var/lib/kubelet/pods/fae6c4b7-98d7-4345-9c19-c42d18c90729/volumes"
Oct 31 19:48:11.638984 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.638953    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:11.670763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.670737    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:11.768950 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.768846    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f9d736-7844-4ddc-861e-007871cfa749-kube-api-access-8n4kg\" (UniqueName: \"kubernetes.io/projected/a5f9d736-7844-4ddc-861e-007871cfa749-kube-api-access-8n4kg\") pod \"a5f9d736-7844-4ddc-861e-007871cfa749\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") "
Oct 31 19:48:11.768950 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.768933    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-openshift-ca\") pod \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") "
Oct 31 19:48:11.769510 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.768990    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-openshift-ca\") pod \"a5f9d736-7844-4ddc-861e-007871cfa749\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") "
Oct 31 19:48:11.769510 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.769079    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-xks-ca\") pod \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") "
Oct 31 19:48:11.769510 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.769182    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-kube-api-access-gkbn5\" (UniqueName: \"kubernetes.io/projected/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-kube-api-access-gkbn5\") pod \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\" (UID: \"196e332b-96d9-4f71-98ee-3ca1da1d5eb5\") "
Oct 31 19:48:11.769510 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.769306    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-xks-ca\") pod \"a5f9d736-7844-4ddc-861e-007871cfa749\" (UID: \"a5f9d736-7844-4ddc-861e-007871cfa749\") "
Oct 31 19:48:11.769510 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.769361    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-openshift-ca" pod "196e332b-96d9-4f71-98ee-3ca1da1d5eb5" (UID: "196e332b-96d9-4f71-98ee-3ca1da1d5eb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:11.770046 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.770021    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-openshift-ca" pod "a5f9d736-7844-4ddc-861e-007871cfa749" (UID: "a5f9d736-7844-4ddc-861e-007871cfa749"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:11.771043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.771018    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.771043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.771044    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.772467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.772438    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-kube-api-access-gkbn5" pod "196e332b-96d9-4f71-98ee-3ca1da1d5eb5" (UID: "196e332b-96d9-4f71-98ee-3ca1da1d5eb5"). InnerVolumeSpecName "kube-api-access-gkbn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:11.772564 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.772460    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9d736-7844-4ddc-861e-007871cfa749-kube-api-access-8n4kg" pod "a5f9d736-7844-4ddc-861e-007871cfa749" (UID: "a5f9d736-7844-4ddc-861e-007871cfa749"). InnerVolumeSpecName "kube-api-access-8n4kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:11.772564 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.772466    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-xks-ca" pod "196e332b-96d9-4f71-98ee-3ca1da1d5eb5" (UID: "196e332b-96d9-4f71-98ee-3ca1da1d5eb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:11.773689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.773638    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-xks-ca" pod "a5f9d736-7844-4ddc-861e-007871cfa749" (UID: "a5f9d736-7844-4ddc-861e-007871cfa749"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:11.873072 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.873038    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkbn5\" (UniqueName: \"kubernetes.io/projected/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-kube-api-access-gkbn5\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.873072 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.873071    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f9d736-7844-4ddc-861e-007871cfa749-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.873295 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.873088    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n4kg\" (UniqueName: \"kubernetes.io/projected/a5f9d736-7844-4ddc-861e-007871cfa749-kube-api-access-8n4kg\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.873295 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:11.873102    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196e332b-96d9-4f71-98ee-3ca1da1d5eb5-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:12.430466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:12.430433    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"
Oct 31 19:48:12.430703 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:12.430434    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd" event={"ID":"196e332b-96d9-4f71-98ee-3ca1da1d5eb5","Type":"ContainerDied","Data":"41f04bb60079eaeb7f295e202532b6f1c748ccab75b6bbe731fef39e699e20e9"}
Oct 31 19:48:12.430703 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:12.430605    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41f04bb60079eaeb7f295e202532b6f1c748ccab75b6bbe731fef39e699e20e9"
Oct 31 19:48:12.432019 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:12.431992    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77" event={"ID":"a5f9d736-7844-4ddc-861e-007871cfa749","Type":"ContainerDied","Data":"3786bfd064295da99b82ffedcb05ffb06ee6c1b88e52d5528e6542e97bcd0a20"}
Oct 31 19:48:12.432127 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:12.432039    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3786bfd064295da99b82ffedcb05ffb06ee6c1b88e52d5528e6542e97bcd0a20"
Oct 31 19:48:12.432127 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:12.432044    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"
Oct 31 19:48:16.060571 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:16.060531    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"]
Oct 31 19:48:16.083903 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:16.083855    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:16.098646 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:16.098602    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d787d57c-lcv9z"]
Oct 31 19:48:17.599753 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.599719    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e380ee2-2c4c-4529-aba0-f5de7368c1f3" path="/var/lib/kubelet/pods/1e380ee2-2c4c-4529-aba0-f5de7368c1f3/volumes"
Oct 31 19:48:17.733789 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.733745    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"]
Oct 31 19:48:17.734475 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734455    5006 cpu_manager.go:414] "containerMap: removing container" podUID="99e14978-4d67-42e1-93a6-268a6deedd69" containerName="revoke-keys"
Oct 31 19:48:17.734577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734476    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e14978-4d67-42e1-93a6-268a6deedd69" containerName="revoke-keys"
Oct 31 19:48:17.734577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734496    5006 cpu_manager.go:414] "containerMap: removing container" podUID="a5f9d736-7844-4ddc-861e-007871cfa749" containerName="revoke-keys"
Oct 31 19:48:17.734577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734504    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f9d736-7844-4ddc-861e-007871cfa749" containerName="revoke-keys"
Oct 31 19:48:17.734577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734518    5006 cpu_manager.go:414] "containerMap: removing container" podUID="196e332b-96d9-4f71-98ee-3ca1da1d5eb5" containerName="revoke-keys"
Oct 31 19:48:17.734577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734526    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196e332b-96d9-4f71-98ee-3ca1da1d5eb5" containerName="revoke-keys"
Oct 31 19:48:17.734577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734564    5006 cpu_manager.go:414] "containerMap: removing container" podUID="777f4f51-efa9-4be1-a86f-626692778040" containerName="revoke-keys"
Oct 31 19:48:17.734577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734571    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777f4f51-efa9-4be1-a86f-626692778040" containerName="revoke-keys"
Oct 31 19:48:17.734975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734763    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e14978-4d67-42e1-93a6-268a6deedd69" containerName="revoke-keys"
Oct 31 19:48:17.734975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734780    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f9d736-7844-4ddc-861e-007871cfa749" containerName="revoke-keys"
Oct 31 19:48:17.734975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734792    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="196e332b-96d9-4f71-98ee-3ca1da1d5eb5" containerName="revoke-keys"
Oct 31 19:48:17.734975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.734808    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="777f4f51-efa9-4be1-a86f-626692778040" containerName="revoke-keys"
Oct 31 19:48:17.739874 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.739847    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:17.796865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.796828    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"]
Oct 31 19:48:17.840035 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.840001    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:48:17.840228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.840001    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:48:17.840228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.840036    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:48:17.842712 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.842689    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:48:17.866629 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.866541    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x5sw\" (UniqueName: \"kubernetes.io/projected/80d1d2b2-736d-4566-8851-7d8e296aeb22-kube-api-access-2x5sw\") pod \"maas-api-revoke-sub-dc83de9b-nsd42\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:17.866629 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.866596    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc83de9b-nsd42\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:17.866907 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.866815    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc83de9b-nsd42\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:17.967962 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.967917    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc83de9b-nsd42\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:17.968190 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.968065    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5sw\" (UniqueName: \"kubernetes.io/projected/80d1d2b2-736d-4566-8851-7d8e296aeb22-kube-api-access-2x5sw\") pod \"maas-api-revoke-sub-dc83de9b-nsd42\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:17.968190 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.968101    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc83de9b-nsd42\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:17.968624 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.968596    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc83de9b-nsd42\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:17.970747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:17.970721    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc83de9b-nsd42\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:18.026694 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:18.026636    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x5sw\" (UniqueName: \"kubernetes.io/projected/80d1d2b2-736d-4566-8851-7d8e296aeb22-kube-api-access-2x5sw\") pod \"maas-api-revoke-sub-dc83de9b-nsd42\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:18.055575 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:18.055540    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:18.430766 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:18.430730    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d1d2b2_736d_4566_8851_7d8e296aeb22.slice/crio-bcdab0ada4d4be953dae3f9ec5f99520080a54020520cbd50fdaa6e208bc1c7d WatchSource:0}: Error finding container bcdab0ada4d4be953dae3f9ec5f99520080a54020520cbd50fdaa6e208bc1c7d: Status 404 returned error can't find the container with id bcdab0ada4d4be953dae3f9ec5f99520080a54020520cbd50fdaa6e208bc1c7d
Oct 31 19:48:18.433962 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:18.433929    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"]
Oct 31 19:48:18.464777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:18.464728    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"]
Oct 31 19:48:18.465038 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:18.464997    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerName="istio-proxy" containerID="cri-o://6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d" gracePeriod=30
Oct 31 19:48:18.469800 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:18.469772    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"]
Oct 31 19:48:18.470060 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:18.469980    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy" containerID="cri-o://eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497" gracePeriod=30
Oct 31 19:48:18.484689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:18.484638    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"]
Oct 31 19:48:18.484944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:18.484912    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerName="istio-proxy" containerID="cri-o://98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f" gracePeriod=30
Oct 31 19:48:18.489419 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:18.489383    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42" event={"ID":"80d1d2b2-736d-4566-8851-7d8e296aeb22","Type":"ContainerStarted","Data":"bcdab0ada4d4be953dae3f9ec5f99520080a54020520cbd50fdaa6e208bc1c7d"}
Oct 31 19:48:19.500547 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.500510    5006 generic.go:356] "Generic (PLEG): container finished" podID="80d1d2b2-736d-4566-8851-7d8e296aeb22" containerID="494e260579a722d9ab0a85ca0553c0ea3fae3926459b523eec8eabb041e3e94e" exitCode=0
Oct 31 19:48:19.501020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.500593    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42" event={"ID":"80d1d2b2-736d-4566-8851-7d8e296aeb22","Type":"ContainerDied","Data":"494e260579a722d9ab0a85ca0553c0ea3fae3926459b523eec8eabb041e3e94e"}
Oct 31 19:48:19.703355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.703263    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"]
Oct 31 19:48:19.707922 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.707897    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.796991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.795690    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.796991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.795800    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj68f\" (UniqueName: \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-kube-api-access-vj68f\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.796991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.795928    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.796991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.795956    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.796991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.796021    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba6c1e9-986e-44e5-a460-030a11eacb91-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.796991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.796084    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.796991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.796113    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.796991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.796186    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.796991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.796298    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.799296 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.799262    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"]
Oct 31 19:48:19.899142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.898584    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.899142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.898710    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj68f\" (UniqueName: \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-kube-api-access-vj68f\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.899142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.898849    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.899142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.898875    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.899142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.898941    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba6c1e9-986e-44e5-a460-030a11eacb91-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.899142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.898991    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.899142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.899020    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.899142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.899099    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.899754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.899225    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.899754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.899703    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.899936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.899913    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.902531 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.902499    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.903267 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.903186    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba6c1e9-986e-44e5-a460-030a11eacb91-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.903451 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.903432    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.903551 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.903526    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.909142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.909103    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.916981 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.916943    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6jhdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:48:19.917151 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.917003    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:48:19.957522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.957441    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:19.958004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:19.957983    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj68f\" (UniqueName: \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-kube-api-access-vj68f\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-b8zss\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:20.023745 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.023699    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:20.462772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.462728    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"]
Oct 31 19:48:20.490988 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:20.490948    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba6c1e9_986e_44e5_a460_030a11eacb91.slice/crio-b79b4e60df0e6c8c7f2dd056a5ca6c2a95c781dd70e11d13477876e5214ece08 WatchSource:0}: Error finding container b79b4e60df0e6c8c7f2dd056a5ca6c2a95c781dd70e11d13477876e5214ece08: Status 404 returned error can't find the container with id b79b4e60df0e6c8c7f2dd056a5ca6c2a95c781dd70e11d13477876e5214ece08
Oct 31 19:48:20.494418 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.494381    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:48:20.494511 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.494452    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:48:20.494511 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.494484    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:48:20.519748 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.519710    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" event={"ID":"aba6c1e9-986e-44e5-a460-030a11eacb91","Type":"ContainerStarted","Data":"b79b4e60df0e6c8c7f2dd056a5ca6c2a95c781dd70e11d13477876e5214ece08"}
Oct 31 19:48:20.696129 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.696099    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:20.813727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.813474    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-openshift-ca\") pod \"80d1d2b2-736d-4566-8851-7d8e296aeb22\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") "
Oct 31 19:48:20.813909 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.813793    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-xks-ca\") pod \"80d1d2b2-736d-4566-8851-7d8e296aeb22\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") "
Oct 31 19:48:20.813909 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.813834    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d1d2b2-736d-4566-8851-7d8e296aeb22-kube-api-access-2x5sw\" (UniqueName: \"kubernetes.io/projected/80d1d2b2-736d-4566-8851-7d8e296aeb22-kube-api-access-2x5sw\") pod \"80d1d2b2-736d-4566-8851-7d8e296aeb22\" (UID: \"80d1d2b2-736d-4566-8851-7d8e296aeb22\") "
Oct 31 19:48:20.817605 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.814603    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-openshift-ca" pod "80d1d2b2-736d-4566-8851-7d8e296aeb22" (UID: "80d1d2b2-736d-4566-8851-7d8e296aeb22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:20.819733 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.819642    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d1d2b2-736d-4566-8851-7d8e296aeb22-kube-api-access-2x5sw" pod "80d1d2b2-736d-4566-8851-7d8e296aeb22" (UID: "80d1d2b2-736d-4566-8851-7d8e296aeb22"). InnerVolumeSpecName "kube-api-access-2x5sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:20.821192 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.821157    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-xks-ca" pod "80d1d2b2-736d-4566-8851-7d8e296aeb22" (UID: "80d1d2b2-736d-4566-8851-7d8e296aeb22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:20.915520 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.915483    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:20.915520 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.915519    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x5sw\" (UniqueName: \"kubernetes.io/projected/80d1d2b2-736d-4566-8851-7d8e296aeb22-kube-api-access-2x5sw\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:20.915947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:20.915533    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d1d2b2-736d-4566-8851-7d8e296aeb22-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:21.041342 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:21.041303    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vxw57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:48:21.041525 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:21.041365    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:48:21.073257 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:21.073160    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:21.534114 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:21.534081    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"
Oct 31 19:48:21.534114 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:21.534097    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42" event={"ID":"80d1d2b2-736d-4566-8851-7d8e296aeb22","Type":"ContainerDied","Data":"bcdab0ada4d4be953dae3f9ec5f99520080a54020520cbd50fdaa6e208bc1c7d"}
Oct 31 19:48:21.534668 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:21.534176    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcdab0ada4d4be953dae3f9ec5f99520080a54020520cbd50fdaa6e208bc1c7d"
Oct 31 19:48:21.536614 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:21.536568    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" event={"ID":"aba6c1e9-986e-44e5-a460-030a11eacb91","Type":"ContainerStarted","Data":"2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7"}
Oct 31 19:48:21.655401 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:21.655342    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" podStartSLOduration=2.655325506 podStartE2EDuration="2.655325506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:21.652614861 +0000 UTC m=+1920.182148933" watchObservedRunningTime="2026-10-31 19:48:21.655325506 +0000 UTC m=+1920.184859558"
Oct 31 19:48:22.020678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.020152    5006 scope.go:122] "RemoveContainer" containerID="140354c81a6764c63c08b68b482df36fe8ce4c887a1c3b09afa82f62032c772a"
Oct 31 19:48:22.025230 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.024401    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:22.027850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.027178    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:48:22.027850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.027247    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 19:48:22.040071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.040037    5006 scope.go:122] "RemoveContainer" containerID="ecd3d59ded8246a66db5dcb7a6e03866e458aefb4fab314821ca9e20d5ad8406"
Oct 31 19:48:22.052203 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.052166    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"]
Oct 31 19:48:22.053078 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.053053    5006 scope.go:122] "RemoveContainer" containerID="2d408947baf4f37504ef4cd20b2389c3ea5aaf965cdd2f540150f2f183bfebae"
Oct 31 19:48:22.058595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.058536    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ba2503-x6lqh"]
Oct 31 19:48:22.064162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.064128    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"]
Oct 31 19:48:22.067960 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.067939    5006 scope.go:122] "RemoveContainer" containerID="940c5f0bc55803b055c46ceb50bc96466cc5c530760e73ad87476e6e94c27408"
Oct 31 19:48:22.070544 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.070518    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c55924-rqbgs"]
Oct 31 19:48:22.079241 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.079218    5006 scope.go:122] "RemoveContainer" containerID="6447e4ea3a665377584d15bc462d2eefb8a41b0b593c2f1615c199cb371036b8"
Oct 31 19:48:22.433369 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:22.433316    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:23.024277 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.024238    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:48:23.024815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.024316    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 19:48:23.613899 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.613854    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc26859a-5db3-49c9-a1b7-68ff3b07ed11" path="/var/lib/kubelet/pods/dc26859a-5db3-49c9-a1b7-68ff3b07ed11/volumes"
Oct 31 19:48:23.614957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.614928    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb9545d-adef-490b-974f-b28ebb9076ac" path="/var/lib/kubelet/pods/fbb9545d-adef-490b-974f-b28ebb9076ac/volumes"
Oct 31 19:48:23.936641 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.936184    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:48:23.964993 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.964929    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:48:23.969020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.968987    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:48:23.982870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.982826    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-podinfo\") pod \"683ca9eb-8f9e-44db-b22b-15cc698948d2\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") "
Oct 31 19:48:23.983078 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.982942    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-socket\") pod \"683ca9eb-8f9e-44db-b22b-15cc698948d2\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") "
Oct 31 19:48:23.983078 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983016    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-token\" (UniqueName: \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-token\") pod \"683ca9eb-8f9e-44db-b22b-15cc698948d2\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") "
Oct 31 19:48:23.983222 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983161    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683ca9eb-8f9e-44db-b22b-15cc698948d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683ca9eb-8f9e-44db-b22b-15cc698948d2-istiod-ca-cert\") pod \"683ca9eb-8f9e-44db-b22b-15cc698948d2\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") "
Oct 31 19:48:23.983289 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983251    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-credential-socket\") pod \"683ca9eb-8f9e-44db-b22b-15cc698948d2\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") "
Oct 31 19:48:23.983350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983292    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-kube-api-access-47wd2\" (UniqueName: \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-kube-api-access-47wd2\") pod \"683ca9eb-8f9e-44db-b22b-15cc698948d2\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") "
Oct 31 19:48:23.983406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983378    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-data\") pod \"683ca9eb-8f9e-44db-b22b-15cc698948d2\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") "
Oct 31 19:48:23.983466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983407    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-certs\") pod \"683ca9eb-8f9e-44db-b22b-15cc698948d2\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") "
Oct 31 19:48:23.983527 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983487    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-envoy\") pod \"683ca9eb-8f9e-44db-b22b-15cc698948d2\" (UID: \"683ca9eb-8f9e-44db-b22b-15cc698948d2\") "
Oct 31 19:48:23.983588 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983544    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-credential-socket" pod "683ca9eb-8f9e-44db-b22b-15cc698948d2" (UID: "683ca9eb-8f9e-44db-b22b-15cc698948d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:23.983663 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983609    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683ca9eb-8f9e-44db-b22b-15cc698948d2-istiod-ca-cert" pod "683ca9eb-8f9e-44db-b22b-15cc698948d2" (UID: "683ca9eb-8f9e-44db-b22b-15cc698948d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:23.984007 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983788    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-data" pod "683ca9eb-8f9e-44db-b22b-15cc698948d2" (UID: "683ca9eb-8f9e-44db-b22b-15cc698948d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:23.984007 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.983972    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-socket" pod "683ca9eb-8f9e-44db-b22b-15cc698948d2" (UID: "683ca9eb-8f9e-44db-b22b-15cc698948d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:23.984595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.984338    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-certs" pod "683ca9eb-8f9e-44db-b22b-15cc698948d2" (UID: "683ca9eb-8f9e-44db-b22b-15cc698948d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:23.984595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.984441    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:23.984595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.984464    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683ca9eb-8f9e-44db-b22b-15cc698948d2-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:23.984595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.984481    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:23.984595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.984495    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:23.986988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.986952    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-kube-api-access-47wd2" pod "683ca9eb-8f9e-44db-b22b-15cc698948d2" (UID: "683ca9eb-8f9e-44db-b22b-15cc698948d2"). InnerVolumeSpecName "kube-api-access-47wd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:23.986988 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.986973    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-podinfo" pod "683ca9eb-8f9e-44db-b22b-15cc698948d2" (UID: "683ca9eb-8f9e-44db-b22b-15cc698948d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:23.987251 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.986989    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-token" pod "683ca9eb-8f9e-44db-b22b-15cc698948d2" (UID: "683ca9eb-8f9e-44db-b22b-15cc698948d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:23.987321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:23.987264    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-envoy" pod "683ca9eb-8f9e-44db-b22b-15cc698948d2" (UID: "683ca9eb-8f9e-44db-b22b-15cc698948d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.024224 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.024179    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:48:24.024433 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.024244    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 19:48:24.085333 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085249    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-data\") pod \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") "
Oct 31 19:48:24.085333 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085303    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-credential-socket\") pod \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") "
Oct 31 19:48:24.085552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085354    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31b98296-1596-4837-8587-18e6ca94599b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b98296-1596-4837-8587-18e6ca94599b-istio-podinfo\") pod \"31b98296-1596-4837-8587-18e6ca94599b\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") "
Oct 31 19:48:24.085552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085387    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istiod-ca-cert\") pod \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") "
Oct 31 19:48:24.085552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085447    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-kube-api-access-2fqt8\" (UniqueName: \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-kube-api-access-2fqt8\") pod \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") "
Oct 31 19:48:24.085552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085484    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-podinfo\") pod \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") "
Oct 31 19:48:24.085802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085567    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-socket\") pod \"31b98296-1596-4837-8587-18e6ca94599b\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") "
Oct 31 19:48:24.085802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085601    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-data" pod "00fa87d5-f9ce-4a19-a70f-fea496c8c40d" (UID: "00fa87d5-f9ce-4a19-a70f-fea496c8c40d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.085802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085614    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-kube-api-access-hbzwm\" (UniqueName: \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-kube-api-access-hbzwm\") pod \"31b98296-1596-4837-8587-18e6ca94599b\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") "
Oct 31 19:48:24.085802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085701    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-credential-socket" pod "00fa87d5-f9ce-4a19-a70f-fea496c8c40d" (UID: "00fa87d5-f9ce-4a19-a70f-fea496c8c40d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.085802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085787    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-certs\") pod \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") "
Oct 31 19:48:24.086081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085828    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-envoy\") pod \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") "
Oct 31 19:48:24.086081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085886    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-token\" (UniqueName: \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-token\") pod \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") "
Oct 31 19:48:24.086081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085896    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-socket" pod "31b98296-1596-4837-8587-18e6ca94599b" (UID: "31b98296-1596-4837-8587-18e6ca94599b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.086081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085903    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istiod-ca-cert" pod "00fa87d5-f9ce-4a19-a70f-fea496c8c40d" (UID: "00fa87d5-f9ce-4a19-a70f-fea496c8c40d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:24.086081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.085971    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-socket\") pod \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\" (UID: \"00fa87d5-f9ce-4a19-a70f-fea496c8c40d\") "
Oct 31 19:48:24.086081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.086008    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-envoy\") pod \"31b98296-1596-4837-8587-18e6ca94599b\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") "
Oct 31 19:48:24.086081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.086026    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-certs" pod "00fa87d5-f9ce-4a19-a70f-fea496c8c40d" (UID: "00fa87d5-f9ce-4a19-a70f-fea496c8c40d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.086081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.086035    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-data\") pod \"31b98296-1596-4837-8587-18e6ca94599b\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") "
Oct 31 19:48:24.086466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.086089    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-credential-socket\") pod \"31b98296-1596-4837-8587-18e6ca94599b\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") "
Oct 31 19:48:24.086466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.086129    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b98296-1596-4837-8587-18e6ca94599b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b98296-1596-4837-8587-18e6ca94599b-istiod-ca-cert\") pod \"31b98296-1596-4837-8587-18e6ca94599b\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") "
Oct 31 19:48:24.086466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.086207    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-certs\") pod \"31b98296-1596-4837-8587-18e6ca94599b\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") "
Oct 31 19:48:24.086466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.086222    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-data" pod "31b98296-1596-4837-8587-18e6ca94599b" (UID: "31b98296-1596-4837-8587-18e6ca94599b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.086466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.086325    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-istio-token\" (UniqueName: \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-istio-token\") pod \"31b98296-1596-4837-8587-18e6ca94599b\" (UID: \"31b98296-1596-4837-8587-18e6ca94599b\") "
Oct 31 19:48:24.086466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.086416    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-certs" pod "31b98296-1596-4837-8587-18e6ca94599b" (UID: "31b98296-1596-4837-8587-18e6ca94599b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.086820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.086609    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-credential-socket" pod "31b98296-1596-4837-8587-18e6ca94599b" (UID: "31b98296-1596-4837-8587-18e6ca94599b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.087243 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087216    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47wd2\" (UniqueName: \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-kube-api-access-47wd2\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087248    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087264    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087281    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087296    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087309    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087324    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087341 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087337    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087351    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087366    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683ca9eb-8f9e-44db-b22b-15cc698948d2-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087380    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087393    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087407    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.087777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.087767    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b98296-1596-4837-8587-18e6ca94599b-istiod-ca-cert" pod "31b98296-1596-4837-8587-18e6ca94599b" (UID: "31b98296-1596-4837-8587-18e6ca94599b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:24.088115 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.088091    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-socket" pod "00fa87d5-f9ce-4a19-a70f-fea496c8c40d" (UID: "00fa87d5-f9ce-4a19-a70f-fea496c8c40d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.088243 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.088222    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-podinfo" pod "00fa87d5-f9ce-4a19-a70f-fea496c8c40d" (UID: "00fa87d5-f9ce-4a19-a70f-fea496c8c40d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:24.088494 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.088468    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-kube-api-access-hbzwm" pod "31b98296-1596-4837-8587-18e6ca94599b" (UID: "31b98296-1596-4837-8587-18e6ca94599b"). InnerVolumeSpecName "kube-api-access-hbzwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:24.088853 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.088824    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-envoy" pod "00fa87d5-f9ce-4a19-a70f-fea496c8c40d" (UID: "00fa87d5-f9ce-4a19-a70f-fea496c8c40d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.088979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.088872    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-token" pod "00fa87d5-f9ce-4a19-a70f-fea496c8c40d" (UID: "00fa87d5-f9ce-4a19-a70f-fea496c8c40d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:24.089057 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.089014    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31b98296-1596-4837-8587-18e6ca94599b-istio-podinfo" pod "31b98296-1596-4837-8587-18e6ca94599b" (UID: "31b98296-1596-4837-8587-18e6ca94599b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:24.089193 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.089173    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-istio-token" pod "31b98296-1596-4837-8587-18e6ca94599b" (UID: "31b98296-1596-4837-8587-18e6ca94599b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:24.089746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.089726    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-kube-api-access-2fqt8" pod "00fa87d5-f9ce-4a19-a70f-fea496c8c40d" (UID: "00fa87d5-f9ce-4a19-a70f-fea496c8c40d"). InnerVolumeSpecName "kube-api-access-2fqt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:24.090267 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.090243    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-envoy" pod "31b98296-1596-4837-8587-18e6ca94599b" (UID: "31b98296-1596-4837-8587-18e6ca94599b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:24.189776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.189735    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbzwm\" (UniqueName: \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-kube-api-access-hbzwm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.189776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.189778    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.190064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.189796    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.190064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.189810    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.190064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.189825    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b98296-1596-4837-8587-18e6ca94599b-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.190064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.189840    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b98296-1596-4837-8587-18e6ca94599b-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.190064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.189853    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b98296-1596-4837-8587-18e6ca94599b-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.190064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.189866    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b98296-1596-4837-8587-18e6ca94599b-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.190064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.189882    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fqt8\" (UniqueName: \"kubernetes.io/projected/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-kube-api-access-2fqt8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.190064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.189895    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fa87d5-f9ce-4a19-a70f-fea496c8c40d-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:24.530549 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.530496    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"]
Oct 31 19:48:24.531465 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531445    5006 cpu_manager.go:414] "containerMap: removing container" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy"
Oct 31 19:48:24.531585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531465    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy"
Oct 31 19:48:24.531585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531545    5006 cpu_manager.go:414] "containerMap: removing container" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerName="istio-proxy"
Oct 31 19:48:24.531585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531554    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerName="istio-proxy"
Oct 31 19:48:24.531585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531570    5006 cpu_manager.go:414] "containerMap: removing container" podUID="80d1d2b2-736d-4566-8851-7d8e296aeb22" containerName="revoke-keys"
Oct 31 19:48:24.531585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531578    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d1d2b2-736d-4566-8851-7d8e296aeb22" containerName="revoke-keys"
Oct 31 19:48:24.531585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531588    5006 cpu_manager.go:414] "containerMap: removing container" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerName="istio-proxy"
Oct 31 19:48:24.531815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531595    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerName="istio-proxy"
Oct 31 19:48:24.531815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531766    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerName="istio-proxy"
Oct 31 19:48:24.531815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531777    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy"
Oct 31 19:48:24.531815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531787    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d1d2b2-736d-4566-8851-7d8e296aeb22" containerName="revoke-keys"
Oct 31 19:48:24.531815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531796    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerName="istio-proxy"
Oct 31 19:48:24.531815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.531812    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy"
Oct 31 19:48:24.537228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.537196    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.576682 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.576616    5006 generic.go:356] "Generic (PLEG): container finished" podID="683ca9eb-8f9e-44db-b22b-15cc698948d2" containerID="6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d" exitCode=0
Oct 31 19:48:24.576928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.576705    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"
Oct 31 19:48:24.576928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.576723    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" event={"ID":"683ca9eb-8f9e-44db-b22b-15cc698948d2","Type":"ContainerDied","Data":"6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d"}
Oct 31 19:48:24.576928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.576795    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc" event={"ID":"683ca9eb-8f9e-44db-b22b-15cc698948d2","Type":"ContainerDied","Data":"612edd9e623744fb3e3598332f026af925e12994f70b1faef3419a6683fb4cfe"}
Oct 31 19:48:24.576928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.576829    5006 scope.go:122] "RemoveContainer" containerID="6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d"
Oct 31 19:48:24.579639 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.578977    5006 generic.go:356] "Generic (PLEG): container finished" podID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" containerID="98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f" exitCode=0
Oct 31 19:48:24.579639 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.579258    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" event={"ID":"00fa87d5-f9ce-4a19-a70f-fea496c8c40d","Type":"ContainerDied","Data":"98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f"}
Oct 31 19:48:24.579639 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.579389    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"
Oct 31 19:48:24.581752 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.581704    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57" event={"ID":"00fa87d5-f9ce-4a19-a70f-fea496c8c40d","Type":"ContainerDied","Data":"b151c4f3c4c9b7b4c8be4db096ee47ce451712f70683aab42e20ae45e92c2b72"}
Oct 31 19:48:24.587234 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.587158    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"]
Oct 31 19:48:24.588779 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.588740    5006 generic.go:356] "Generic (PLEG): container finished" podID="31b98296-1596-4837-8587-18e6ca94599b" containerID="eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497" exitCode=0
Oct 31 19:48:24.588918 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.588826    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"
Oct 31 19:48:24.589088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.588990    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" event={"ID":"31b98296-1596-4837-8587-18e6ca94599b","Type":"ContainerDied","Data":"eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497"}
Oct 31 19:48:24.589088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.589054    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646" event={"ID":"31b98296-1596-4837-8587-18e6ca94599b","Type":"ContainerDied","Data":"fe8af90211ce357c51271c43b608ba6f327665841cff41237b0ca6fbfc7a1667"}
Oct 31 19:48:24.590997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.590756    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:48:24.590997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.590773    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:48:24.590997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.590826    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:48:24.590997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.590969    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:48:24.594812 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.594789    5006 scope.go:122] "RemoveContainer" containerID="6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d"
Oct 31 19:48:24.596944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.595927    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c135fa-xh2h9\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.596944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.595988    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxl5r\" (UniqueName: \"kubernetes.io/projected/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-kube-api-access-wxl5r\") pod \"maas-api-revoke-sub-00c135fa-xh2h9\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.596944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.596048    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c135fa-xh2h9\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.596944 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:48:24.596674    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d\": container with ID starting with 6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d not found: ID does not exist" containerID="6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d"
Oct 31 19:48:24.596944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.596710    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d"} err="failed to get container status \"6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d\": rpc error: code = NotFound desc = could not find container \"6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d\": container with ID starting with 6c5d9663ce8b4086d9a2ffb61dc96d15c94479d2edb571d0f7cfb11fbad14c2d not found: ID does not exist"
Oct 31 19:48:24.596944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.596736    5006 scope.go:122] "RemoveContainer" containerID="98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f"
Oct 31 19:48:24.635414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.635392    5006 scope.go:122] "RemoveContainer" containerID="98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f"
Oct 31 19:48:24.635857 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:48:24.635834    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f\": container with ID starting with 98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f not found: ID does not exist" containerID="98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f"
Oct 31 19:48:24.635984 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.635963    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f"} err="failed to get container status \"98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f\": rpc error: code = NotFound desc = could not find container \"98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f\": container with ID starting with 98e49228f06b4f5137ec6d508f41b79daa5beab8f630915ef0e47392ccbecc8f not found: ID does not exist"
Oct 31 19:48:24.636046 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.636037    5006 scope.go:122] "RemoveContainer" containerID="eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497"
Oct 31 19:48:24.651551 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.651522    5006 scope.go:122] "RemoveContainer" containerID="4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288"
Oct 31 19:48:24.665285 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.665262    5006 scope.go:122] "RemoveContainer" containerID="eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497"
Oct 31 19:48:24.665760 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:48:24.665732    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497\": container with ID starting with eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497 not found: ID does not exist" containerID="eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497"
Oct 31 19:48:24.665817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.665773    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497"} err="failed to get container status \"eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497\": rpc error: code = NotFound desc = could not find container \"eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497\": container with ID starting with eb6642856d3614018f5fa9588d138af5f466801ae614d46efe657fddced24497 not found: ID does not exist"
Oct 31 19:48:24.665817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.665797    5006 scope.go:122] "RemoveContainer" containerID="4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288"
Oct 31 19:48:24.666115 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:48:24.666097    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288\": container with ID starting with 4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288 not found: ID does not exist" containerID="4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288"
Oct 31 19:48:24.666152 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.666125    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288"} err="failed to get container status \"4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288\": rpc error: code = NotFound desc = could not find container \"4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288\": container with ID starting with 4b5187e3dd90edea9de84cc256e10170728d04a040a9441afc4d0ed51e8cb288 not found: ID does not exist"
Oct 31 19:48:24.721906 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.721870    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c135fa-xh2h9\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.722065 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.721916    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxl5r\" (UniqueName: \"kubernetes.io/projected/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-kube-api-access-wxl5r\") pod \"maas-api-revoke-sub-00c135fa-xh2h9\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.722065 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.721956    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c135fa-xh2h9\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.722805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.722772    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c135fa-xh2h9\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.724615 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.724586    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c135fa-xh2h9\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.782732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.782637    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxl5r\" (UniqueName: \"kubernetes.io/projected/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-kube-api-access-wxl5r\") pod \"maas-api-revoke-sub-00c135fa-xh2h9\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.784228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.784201    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"]
Oct 31 19:48:24.788870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.788838    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vxw57"]
Oct 31 19:48:24.853904 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.853868    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:24.927821 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.926775    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"]
Oct 31 19:48:24.933092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:24.933054    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6jhdc"]
Oct 31 19:48:25.029522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:25.027808    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:48:25.029522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:25.027866    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 19:48:25.045705 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:25.045458    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"]
Oct 31 19:48:25.047306 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:25.047270    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ks646"]
Oct 31 19:48:25.266402 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:25.266359    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982a0aa3_ccc3_4b76_9a99_ba47c45a3f67.slice/crio-04201974e043e3b53611bdedc341081a87836e241fb48c1ad17f2bb2ea1b419d WatchSource:0}: Error finding container 04201974e043e3b53611bdedc341081a87836e241fb48c1ad17f2bb2ea1b419d: Status 404 returned error can't find the container with id 04201974e043e3b53611bdedc341081a87836e241fb48c1ad17f2bb2ea1b419d
Oct 31 19:48:25.271049 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:25.271022    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"]
Oct 31 19:48:25.628927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:25.628890    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fa87d5-f9ce-4a19-a70f-fea496c8c40d" path="/var/lib/kubelet/pods/00fa87d5-f9ce-4a19-a70f-fea496c8c40d/volumes"
Oct 31 19:48:25.629509 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:25.629490    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b98296-1596-4837-8587-18e6ca94599b" path="/var/lib/kubelet/pods/31b98296-1596-4837-8587-18e6ca94599b/volumes"
Oct 31 19:48:25.630048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:25.630032    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683ca9eb-8f9e-44db-b22b-15cc698948d2" path="/var/lib/kubelet/pods/683ca9eb-8f9e-44db-b22b-15cc698948d2/volumes"
Oct 31 19:48:25.630737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:25.630717    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9" event={"ID":"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67","Type":"ContainerStarted","Data":"04201974e043e3b53611bdedc341081a87836e241fb48c1ad17f2bb2ea1b419d"}
Oct 31 19:48:26.024384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.024284    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:48:26.024384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.024345    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 19:48:26.515119 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.515026    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:26.650340 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.650306    5006 generic.go:356] "Generic (PLEG): container finished" podID="982a0aa3-ccc3-4b76-9a99-ba47c45a3f67" containerID="3223338bda25244d81412e8cada002fcd956d0ffbe29c77f4cab996208809614" exitCode=0
Oct 31 19:48:26.651060 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.650521    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9" event={"ID":"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67","Type":"ContainerDied","Data":"3223338bda25244d81412e8cada002fcd956d0ffbe29c77f4cab996208809614"}
Oct 31 19:48:26.678456 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.678372    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"]
Oct 31 19:48:26.679080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.679063    5006 cpu_manager.go:414] "containerMap: removing container" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy"
Oct 31 19:48:26.679159 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.679081    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b98296-1596-4837-8587-18e6ca94599b" containerName="istio-proxy"
Oct 31 19:48:26.683097 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.683075    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:26.738788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.738744    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"]
Oct 31 19:48:26.857560 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.857513    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42247df1-6vv7c\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:26.857771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.857748    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42247df1-6vv7c\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:26.857921 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.857903    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cgrj\" (UniqueName: \"kubernetes.io/projected/48207f5c-b0fe-45b1-9114-5cdef66319ef-kube-api-access-5cgrj\") pod \"maas-api-revoke-sub-42247df1-6vv7c\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:26.959946 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.959846    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42247df1-6vv7c\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:26.960096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.960031    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42247df1-6vv7c\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:26.960176 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.960157    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgrj\" (UniqueName: \"kubernetes.io/projected/48207f5c-b0fe-45b1-9114-5cdef66319ef-kube-api-access-5cgrj\") pod \"maas-api-revoke-sub-42247df1-6vv7c\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:26.960715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.960692    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42247df1-6vv7c\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:26.962625 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:26.962598    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42247df1-6vv7c\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:27.018061 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:27.018026    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cgrj\" (UniqueName: \"kubernetes.io/projected/48207f5c-b0fe-45b1-9114-5cdef66319ef-kube-api-access-5cgrj\") pod \"maas-api-revoke-sub-42247df1-6vv7c\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:27.024188 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:27.024155    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:48:27.024295 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:27.024221    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 19:48:27.310391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:27.310295    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:27.690691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:27.690637    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"]
Oct 31 19:48:27.911092 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:27.911060    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:28.089987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.089945    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-xks-ca\") pod \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") "
Oct 31 19:48:28.091830 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.091797    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-kube-api-access-wxl5r\" (UniqueName: \"kubernetes.io/projected/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-kube-api-access-wxl5r\") pod \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") "
Oct 31 19:48:28.094105 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.094068    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-openshift-ca" pod "982a0aa3-ccc3-4b76-9a99-ba47c45a3f67" (UID: "982a0aa3-ccc3-4b76-9a99-ba47c45a3f67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:28.094365 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.092460    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-openshift-ca\") pod \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\" (UID: \"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67\") "
Oct 31 19:48:28.096268 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.096227    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:28.105711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.101981    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-kube-api-access-wxl5r" pod "982a0aa3-ccc3-4b76-9a99-ba47c45a3f67" (UID: "982a0aa3-ccc3-4b76-9a99-ba47c45a3f67"). InnerVolumeSpecName "kube-api-access-wxl5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:28.109821 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.109773    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-xks-ca" pod "982a0aa3-ccc3-4b76-9a99-ba47c45a3f67" (UID: "982a0aa3-ccc3-4b76-9a99-ba47c45a3f67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:28.191198 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.191150    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:28.192482 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.192452    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:28.193057 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.193042    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:48:28.198269 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.198246    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:28.198269 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.198277    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxl5r\" (UniqueName: \"kubernetes.io/projected/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67-kube-api-access-wxl5r\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:28.492572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.492532    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"]
Oct 31 19:48:28.493350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.493327    5006 cpu_manager.go:414] "containerMap: removing container" podUID="982a0aa3-ccc3-4b76-9a99-ba47c45a3f67" containerName="revoke-keys"
Oct 31 19:48:28.493350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.493350    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982a0aa3-ccc3-4b76-9a99-ba47c45a3f67" containerName="revoke-keys"
Oct 31 19:48:28.493581 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.493567    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="982a0aa3-ccc3-4b76-9a99-ba47c45a3f67" containerName="revoke-keys"
Oct 31 19:48:28.497688 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.497642    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:28.563054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.563012    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"]
Oct 31 19:48:28.576019 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.575977    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:48:28.603924 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.603884    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jrd\" (UniqueName: \"kubernetes.io/projected/c1d96a6f-628f-4030-9127-0e011cc1eeb5-kube-api-access-h5jrd\") pod \"maas-api-revoke-sub-7dadc90d-54xgm\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:28.604097 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.604037    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dadc90d-54xgm\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:28.604715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.604333    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dadc90d-54xgm\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:28.673404 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.673367    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"
Oct 31 19:48:28.673613 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.673377    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9" event={"ID":"982a0aa3-ccc3-4b76-9a99-ba47c45a3f67","Type":"ContainerDied","Data":"04201974e043e3b53611bdedc341081a87836e241fb48c1ad17f2bb2ea1b419d"}
Oct 31 19:48:28.673613 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.673503    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04201974e043e3b53611bdedc341081a87836e241fb48c1ad17f2bb2ea1b419d"
Oct 31 19:48:28.675433 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.675402    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c" event={"ID":"48207f5c-b0fe-45b1-9114-5cdef66319ef","Type":"ContainerStarted","Data":"4f3667bfacfdcd43b0314837aac1b6af25968f82ca84c52b62503ffef9b1edbc"}
Oct 31 19:48:28.675582 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.675459    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c" event={"ID":"48207f5c-b0fe-45b1-9114-5cdef66319ef","Type":"ContainerStarted","Data":"8c01d15b7d39fe88d40e0098ee06fb63af3f71603517296e3b177b0831dad79b"}
Oct 31 19:48:28.705820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.705781    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jrd\" (UniqueName: \"kubernetes.io/projected/c1d96a6f-628f-4030-9127-0e011cc1eeb5-kube-api-access-h5jrd\") pod \"maas-api-revoke-sub-7dadc90d-54xgm\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:28.706355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.705831    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dadc90d-54xgm\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:28.706355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.706166    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dadc90d-54xgm\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:28.706616 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.706593    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dadc90d-54xgm\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:28.708641 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.708617    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dadc90d-54xgm\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:28.763288 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.763215    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jrd\" (UniqueName: \"kubernetes.io/projected/c1d96a6f-628f-4030-9127-0e011cc1eeb5-kube-api-access-h5jrd\") pod \"maas-api-revoke-sub-7dadc90d-54xgm\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:28.789388 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.789330    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c" podStartSLOduration=2.789305008 podStartE2EDuration="2.789305008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:28.788889653 +0000 UTC m=+1927.318423707" watchObservedRunningTime="2026-10-31 19:48:28.789305008 +0000 UTC m=+1927.318839061"
Oct 31 19:48:28.813802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:28.813772    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:29.305298 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:29.305262    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"]
Oct 31 19:48:29.692671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:29.692462    5006 generic.go:356] "Generic (PLEG): container finished" podID="48207f5c-b0fe-45b1-9114-5cdef66319ef" containerID="4f3667bfacfdcd43b0314837aac1b6af25968f82ca84c52b62503ffef9b1edbc" exitCode=0
Oct 31 19:48:29.692671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:29.692548    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c" event={"ID":"48207f5c-b0fe-45b1-9114-5cdef66319ef","Type":"ContainerDied","Data":"4f3667bfacfdcd43b0314837aac1b6af25968f82ca84c52b62503ffef9b1edbc"}
Oct 31 19:48:29.700642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:29.694259    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm" event={"ID":"c1d96a6f-628f-4030-9127-0e011cc1eeb5","Type":"ContainerStarted","Data":"1721c2d1b948ceb8ffdf5b38ebab89f41b55739308ffc59de7664183e1fdc2e6"}
Oct 31 19:48:30.802815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:30.802758    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:30.827093 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:30.826994    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm" event={"ID":"c1d96a6f-628f-4030-9127-0e011cc1eeb5","Type":"ContainerStarted","Data":"53e738e930b99fd4bd78f6286a28e6081f2f0a86b660c2f796d6a1ce1aecb8ca"}
Oct 31 19:48:30.888704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:30.887991    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.108679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.106719    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"]
Oct 31 19:48:31.163817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.163763    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.186676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.176290    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1a0bbb-f4mmt"]
Oct 31 19:48:31.225675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.225074    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.267677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.257744    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.279678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.269784    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:48:31.279678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.269856    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.315677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.314748    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.330809 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.330755    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.331823 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.331090    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:48:31.357677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.344752    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.382733 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.368756    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.504689 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.499727    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Oct 31 19:48:31.517841 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.517787    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.847728 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.847235    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.850617 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.848404    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:31.913735 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.909756    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:48:31.913735 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:31.910235    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 31 19:48:32.071597 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.071551    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ccf36a-305b-4926-a5d4-5ba5aa59f5fd" path="/var/lib/kubelet/pods/61ccf36a-305b-4926-a5d4-5ba5aa59f5fd/volumes"
Oct 31 19:48:32.074929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.074878    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:48:32.075247 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.075230    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:48:32.169116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.167803    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:32.169116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.168111    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:48:32.288679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.279059    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:32.288679 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.280333    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:48:32.338242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.331303    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:32.338242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.333118    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:48:32.400677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.374099    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:48:32.400677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.375736    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:48:32.400677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.388928    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:32.400677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.389261    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:48:32.538049 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.537942    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:32.538677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.538635    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:48:32.578164 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.577234    5006 generic.go:356] "Generic (PLEG): container finished" podID="c1d96a6f-628f-4030-9127-0e011cc1eeb5" containerID="53e738e930b99fd4bd78f6286a28e6081f2f0a86b660c2f796d6a1ce1aecb8ca" exitCode=0
Oct 31 19:48:32.578164 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:32.577635    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm" event={"ID":"c1d96a6f-628f-4030-9127-0e011cc1eeb5","Type":"ContainerDied","Data":"53e738e930b99fd4bd78f6286a28e6081f2f0a86b660c2f796d6a1ce1aecb8ca"}
Oct 31 19:48:33.370675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.348753    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.408334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.373771    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.408334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.271740    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.408334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.394925    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.411838 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.410805    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.573684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.569085    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.672784 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.647319    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:48:33.673412 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.652329    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.725678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.673592    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:48:33.725678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.704212    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.725678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.681883    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 31 19:48:33.725678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.674434    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:48:33.725678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.699464    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.725678 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.719692    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.739677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.731388    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:33.785684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.762089    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:33.802599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.765644    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:48:33.807982 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.807951    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:48:33.920673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:33.810741    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": context deadline exceeded"
Oct 31 19:48:34.154936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.095543    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:34.204100 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.203915    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:34.204369 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.204343    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-openshift-ca\") pod \"48207f5c-b0fe-45b1-9114-5cdef66319ef\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") "
Oct 31 19:48:34.204663 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.204616    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-xks-ca\") pod \"48207f5c-b0fe-45b1-9114-5cdef66319ef\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") "
Oct 31 19:48:34.205796 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.205770    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48207f5c-b0fe-45b1-9114-5cdef66319ef-kube-api-access-5cgrj\" (UniqueName: \"kubernetes.io/projected/48207f5c-b0fe-45b1-9114-5cdef66319ef-kube-api-access-5cgrj\") pod \"48207f5c-b0fe-45b1-9114-5cdef66319ef\" (UID: \"48207f5c-b0fe-45b1-9114-5cdef66319ef\") "
Oct 31 19:48:34.225253 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.223942    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-openshift-ca" pod "48207f5c-b0fe-45b1-9114-5cdef66319ef" (UID: "48207f5c-b0fe-45b1-9114-5cdef66319ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:34.290676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.286734    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c" event={"ID":"48207f5c-b0fe-45b1-9114-5cdef66319ef","Type":"ContainerDied","Data":"8c01d15b7d39fe88d40e0098ee06fb63af3f71603517296e3b177b0831dad79b"}
Oct 31 19:48:34.290676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.286828    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c01d15b7d39fe88d40e0098ee06fb63af3f71603517296e3b177b0831dad79b"
Oct 31 19:48:34.290676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.286894    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"
Oct 31 19:48:34.337421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.333738    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48207f5c-b0fe-45b1-9114-5cdef66319ef-kube-api-access-5cgrj" pod "48207f5c-b0fe-45b1-9114-5cdef66319ef" (UID: "48207f5c-b0fe-45b1-9114-5cdef66319ef"). InnerVolumeSpecName "kube-api-access-5cgrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:34.337421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.335561    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cgrj\" (UniqueName: \"kubernetes.io/projected/48207f5c-b0fe-45b1-9114-5cdef66319ef-kube-api-access-5cgrj\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:34.337421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.335593    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:34.364844 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.354085    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:34.366870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.366797    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-xks-ca" pod "48207f5c-b0fe-45b1-9114-5cdef66319ef" (UID: "48207f5c-b0fe-45b1-9114-5cdef66319ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:34.400696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.399798    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:34.400696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.400288    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:34.445029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.440008    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48207f5c-b0fe-45b1-9114-5cdef66319ef-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:34.492674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.489433    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 19:48:34.777963 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.647459    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded"
Oct 31 19:48:34.777963 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.648227    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:48:34.777963 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.732945    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:48:34.793516 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.793469    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:48:34.794194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.794159    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:34.794611 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.794584    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:34.795005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.794978    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Oct 31 19:48:34.795367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.795344    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:48:34.902958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.902909    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:34.959798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.903228    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:34.960723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.960697    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:48:34.989936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:34.989887    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:35.303330 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:35.303242    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 19:48:35.375350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:35.375307    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:48:35.395213 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:35.395165    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:35.395531 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:35.385134    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:35.410611 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:35.410569    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:35.443786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:35.443734    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:35.730609 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:35.730559    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:36.005734 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.005099    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 31 19:48:36.014256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.014064    5006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:36.017609 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.014473    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:48:36.026756 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:48:36.026696    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:48:36.155116 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.139271    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:36.176876 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.176838    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 19:48:36.177266 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.177250    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 19:48:36.177523 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.177509    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 19:48:36.177787 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.177745    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 19:48:36.177957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.177946    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 19:48:36.178135 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.178126    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 19:48:36.178290 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.178269    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 19:48:36.181378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.181348    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 19:48:36.194059 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.186502    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 19:48:36.194059 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.186563    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm" event={"ID":"c1d96a6f-628f-4030-9127-0e011cc1eeb5","Type":"ContainerDied","Data":"1721c2d1b948ceb8ffdf5b38ebab89f41b55739308ffc59de7664183e1fdc2e6"}
Oct 31 19:48:36.194059 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.186628    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1721c2d1b948ceb8ffdf5b38ebab89f41b55739308ffc59de7664183e1fdc2e6"
Oct 31 19:48:36.213778 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.213746    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 19:48:36.231067 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.231025    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d96a6f-628f-4030-9127-0e011cc1eeb5-kube-api-access-h5jrd\" (UniqueName: \"kubernetes.io/projected/c1d96a6f-628f-4030-9127-0e011cc1eeb5-kube-api-access-h5jrd\") pod \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") "
Oct 31 19:48:36.232145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.232109    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-openshift-ca\") pod \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") "
Oct 31 19:48:36.232292 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.232234    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-xks-ca\") pod \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\" (UID: \"c1d96a6f-628f-4030-9127-0e011cc1eeb5\") "
Oct 31 19:48:36.237517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.237485    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-openshift-ca" pod "c1d96a6f-628f-4030-9127-0e011cc1eeb5" (UID: "c1d96a6f-628f-4030-9127-0e011cc1eeb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:36.268445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.264916    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"]
Oct 31 19:48:36.268445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.265977    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c1d96a6f-628f-4030-9127-0e011cc1eeb5" containerName="revoke-keys"
Oct 31 19:48:36.268445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.265999    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d96a6f-628f-4030-9127-0e011cc1eeb5" containerName="revoke-keys"
Oct 31 19:48:36.268445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.266031    5006 cpu_manager.go:414] "containerMap: removing container" podUID="48207f5c-b0fe-45b1-9114-5cdef66319ef" containerName="revoke-keys"
Oct 31 19:48:36.268445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.266039    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48207f5c-b0fe-45b1-9114-5cdef66319ef" containerName="revoke-keys"
Oct 31 19:48:36.268445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.266227    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d96a6f-628f-4030-9127-0e011cc1eeb5" containerName="revoke-keys"
Oct 31 19:48:36.268445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.266240    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="48207f5c-b0fe-45b1-9114-5cdef66319ef" containerName="revoke-keys"
Oct 31 19:48:36.272168 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.272047    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d96a6f-628f-4030-9127-0e011cc1eeb5-kube-api-access-h5jrd" pod "c1d96a6f-628f-4030-9127-0e011cc1eeb5" (UID: "c1d96a6f-628f-4030-9127-0e011cc1eeb5"). InnerVolumeSpecName "kube-api-access-h5jrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:36.272168 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.272132    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-xks-ca" pod "c1d96a6f-628f-4030-9127-0e011cc1eeb5" (UID: "c1d96a6f-628f-4030-9127-0e011cc1eeb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:36.276594 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.276565    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"]
Oct 31 19:48:36.276930 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.276912    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:36.334486 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.334429    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpnwv\" (UniqueName: \"kubernetes.io/projected/2d1f2098-72a2-441b-a918-895637f761cf-kube-api-access-jpnwv\") pod \"maas-api-revoke-sub-6cfbfd03-sgjv6\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:36.334687 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.334607    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cfbfd03-sgjv6\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:36.334943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.334922    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cfbfd03-sgjv6\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:36.335373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.335347    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5jrd\" (UniqueName: \"kubernetes.io/projected/c1d96a6f-628f-4030-9127-0e011cc1eeb5-kube-api-access-h5jrd\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:36.335446 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.335377    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:36.335446 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.335391    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d96a6f-628f-4030-9127-0e011cc1eeb5-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:36.436516 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.436479    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpnwv\" (UniqueName: \"kubernetes.io/projected/2d1f2098-72a2-441b-a918-895637f761cf-kube-api-access-jpnwv\") pod \"maas-api-revoke-sub-6cfbfd03-sgjv6\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:36.437241 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.437212    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cfbfd03-sgjv6\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:36.437429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.437412    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cfbfd03-sgjv6\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:36.438082 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.438057    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cfbfd03-sgjv6\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:36.451297 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.451259    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cfbfd03-sgjv6\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:36.513121 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.513044    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpnwv\" (UniqueName: \"kubernetes.io/projected/2d1f2098-72a2-441b-a918-895637f761cf-kube-api-access-jpnwv\") pod \"maas-api-revoke-sub-6cfbfd03-sgjv6\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:36.599111 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:36.599068    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:37.019112 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:37.019039    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"]
Oct 31 19:48:37.020206 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:37.020087    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1f2098_72a2_441b_a918_895637f761cf.slice/crio-06956e1e195ebdb27838b3ef4f572aac782223278670922579d21ededd71f32a WatchSource:0}: Error finding container 06956e1e195ebdb27838b3ef4f572aac782223278670922579d21ededd71f32a: Status 404 returned error can't find the container with id 06956e1e195ebdb27838b3ef4f572aac782223278670922579d21ededd71f32a
Oct 31 19:48:37.196955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:37.196925    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"
Oct 31 19:48:37.199085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:37.199043    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6" event={"ID":"2d1f2098-72a2-441b-a918-895637f761cf","Type":"ContainerStarted","Data":"06956e1e195ebdb27838b3ef4f572aac782223278670922579d21ededd71f32a"}
Oct 31 19:48:37.872080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:37.872046    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"]
Oct 31 19:48:37.876700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:37.876663    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:37.932322 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:37.932283    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"]
Oct 31 19:48:37.966634 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:37.966600    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8z28\" (UniqueName: \"kubernetes.io/projected/6489a436-a0d9-43c5-a21d-414f083ee94f-kube-api-access-p8z28\") pod \"maas-api-revoke-sub-23f86095-mnm6m\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:37.966829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:37.966730    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23f86095-mnm6m\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:37.966951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:37.966929    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23f86095-mnm6m\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:38.068909 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.068645    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8z28\" (UniqueName: \"kubernetes.io/projected/6489a436-a0d9-43c5-a21d-414f083ee94f-kube-api-access-p8z28\") pod \"maas-api-revoke-sub-23f86095-mnm6m\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:38.068909 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.068725    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23f86095-mnm6m\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:38.069052 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.068931    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23f86095-mnm6m\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:38.069795 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.069765    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23f86095-mnm6m\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:38.071348 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.071323    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23f86095-mnm6m\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:38.125827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.125752    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8z28\" (UniqueName: \"kubernetes.io/projected/6489a436-a0d9-43c5-a21d-414f083ee94f-kube-api-access-p8z28\") pod \"maas-api-revoke-sub-23f86095-mnm6m\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:38.201613 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.201580    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:38.207122 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.207089    5006 generic.go:356] "Generic (PLEG): container finished" podID="2d1f2098-72a2-441b-a918-895637f761cf" containerID="05e822b841034854489dea9351e05267e73169da4e9fa5f1a8a151bb41c2e2f4" exitCode=0
Oct 31 19:48:38.207278 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.207181    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6" event={"ID":"2d1f2098-72a2-441b-a918-895637f761cf","Type":"ContainerDied","Data":"05e822b841034854489dea9351e05267e73169da4e9fa5f1a8a151bb41c2e2f4"}
Oct 31 19:48:38.488426 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.488331    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:38.576039 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:38.576005    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6489a436_a0d9_43c5_a21d_414f083ee94f.slice/crio-0ff9639b124bf53b4701377f5cfc0b2106b2fae7572a7f7606d6e5c4906ff18c WatchSource:0}: Error finding container 0ff9639b124bf53b4701377f5cfc0b2106b2fae7572a7f7606d6e5c4906ff18c: Status 404 returned error can't find the container with id 0ff9639b124bf53b4701377f5cfc0b2106b2fae7572a7f7606d6e5c4906ff18c
Oct 31 19:48:38.586976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:38.586938    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"]
Oct 31 19:48:39.218438 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.218391    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m" event={"ID":"6489a436-a0d9-43c5-a21d-414f083ee94f","Type":"ContainerStarted","Data":"d5449d16922fa5decf8fefcb26f44fe324a3f09203691ee20b4e2e745afec5e1"}
Oct 31 19:48:39.218878 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.218458    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m" event={"ID":"6489a436-a0d9-43c5-a21d-414f083ee94f","Type":"ContainerStarted","Data":"0ff9639b124bf53b4701377f5cfc0b2106b2fae7572a7f7606d6e5c4906ff18c"}
Oct 31 19:48:39.242455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.242418    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"]
Oct 31 19:48:39.247099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.247078    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.294773 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.288064    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8wf\" (UniqueName: \"kubernetes.io/projected/9eb48c2a-64c3-493e-bae0-fd287b943cbe-kube-api-access-jk8wf\") pod \"maas-api-revoke-sub-c2a6940a-58xns\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.294773 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.288589    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a6940a-58xns\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.294773 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.288640    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a6940a-58xns\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.305169 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.305128    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"]
Oct 31 19:48:39.339493 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.339426    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m" podStartSLOduration=2.339403277 podStartE2EDuration="2.339403277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:39.331851465 +0000 UTC m=+1937.861385517" watchObservedRunningTime="2026-10-31 19:48:39.339403277 +0000 UTC m=+1937.868937330"
Oct 31 19:48:39.390323 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.390287    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a6940a-58xns\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.390451 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.390343    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a6940a-58xns\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.390451 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.390430    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8wf\" (UniqueName: \"kubernetes.io/projected/9eb48c2a-64c3-493e-bae0-fd287b943cbe-kube-api-access-jk8wf\") pod \"maas-api-revoke-sub-c2a6940a-58xns\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.391115 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.391081    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a6940a-58xns\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.392835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.392813    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a6940a-58xns\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.429236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.429205    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:39.449699 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.449631    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8wf\" (UniqueName: \"kubernetes.io/projected/9eb48c2a-64c3-493e-bae0-fd287b943cbe-kube-api-access-jk8wf\") pod \"maas-api-revoke-sub-c2a6940a-58xns\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.491129 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.491020    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-xks-ca\") pod \"2d1f2098-72a2-441b-a918-895637f761cf\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") "
Oct 31 19:48:39.491299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.491267    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1f2098-72a2-441b-a918-895637f761cf-kube-api-access-jpnwv\" (UniqueName: \"kubernetes.io/projected/2d1f2098-72a2-441b-a918-895637f761cf-kube-api-access-jpnwv\") pod \"2d1f2098-72a2-441b-a918-895637f761cf\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") "
Oct 31 19:48:39.491381 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.491318    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-openshift-ca\") pod \"2d1f2098-72a2-441b-a918-895637f761cf\" (UID: \"2d1f2098-72a2-441b-a918-895637f761cf\") "
Oct 31 19:48:39.491755 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.491724    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-openshift-ca" pod "2d1f2098-72a2-441b-a918-895637f761cf" (UID: "2d1f2098-72a2-441b-a918-895637f761cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:39.492008 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.491986    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:39.493262 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.493232    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-xks-ca" pod "2d1f2098-72a2-441b-a918-895637f761cf" (UID: "2d1f2098-72a2-441b-a918-895637f761cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:39.493504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.493484    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1f2098-72a2-441b-a918-895637f761cf-kube-api-access-jpnwv" pod "2d1f2098-72a2-441b-a918-895637f761cf" (UID: "2d1f2098-72a2-441b-a918-895637f761cf"). InnerVolumeSpecName "kube-api-access-jpnwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:39.563713 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.563680    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:39.580423 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.580377    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:39.593976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.593903    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpnwv\" (UniqueName: \"kubernetes.io/projected/2d1f2098-72a2-441b-a918-895637f761cf-kube-api-access-jpnwv\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:39.593976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.593947    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d1f2098-72a2-441b-a918-895637f761cf-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:39.798075 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.797984    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:39.949184 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:39.949150    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb48c2a_64c3_493e_bae0_fd287b943cbe.slice/crio-4e6178f7ba037a80b9c19a7929163d96ed3b72f44284a6036b1f429e16e8524b WatchSource:0}: Error finding container 4e6178f7ba037a80b9c19a7929163d96ed3b72f44284a6036b1f429e16e8524b: Status 404 returned error can't find the container with id 4e6178f7ba037a80b9c19a7929163d96ed3b72f44284a6036b1f429e16e8524b
Oct 31 19:48:39.954846 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:39.954824    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"]
Oct 31 19:48:40.025843 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.025782    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:40.244013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.243975    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns" event={"ID":"9eb48c2a-64c3-493e-bae0-fd287b943cbe","Type":"ContainerStarted","Data":"4e6178f7ba037a80b9c19a7929163d96ed3b72f44284a6036b1f429e16e8524b"}
Oct 31 19:48:40.246617 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.246591    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"
Oct 31 19:48:40.246798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.246617    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6" event={"ID":"2d1f2098-72a2-441b-a918-895637f761cf","Type":"ContainerDied","Data":"06956e1e195ebdb27838b3ef4f572aac782223278670922579d21ededd71f32a"}
Oct 31 19:48:40.246798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.246684    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06956e1e195ebdb27838b3ef4f572aac782223278670922579d21ededd71f32a"
Oct 31 19:48:40.248837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.248796    5006 generic.go:356] "Generic (PLEG): container finished" podID="6489a436-a0d9-43c5-a21d-414f083ee94f" containerID="d5449d16922fa5decf8fefcb26f44fe324a3f09203691ee20b4e2e745afec5e1" exitCode=0
Oct 31 19:48:40.248965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.248859    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m" event={"ID":"6489a436-a0d9-43c5-a21d-414f083ee94f","Type":"ContainerDied","Data":"d5449d16922fa5decf8fefcb26f44fe324a3f09203691ee20b4e2e745afec5e1"}
Oct 31 19:48:40.254542 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.254516    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11068-57371e"]
Oct 31 19:48:40.255210 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.255196    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2d1f2098-72a2-441b-a918-895637f761cf" containerName="revoke-keys"
Oct 31 19:48:40.255261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.255212    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1f2098-72a2-441b-a918-895637f761cf" containerName="revoke-keys"
Oct 31 19:48:40.255329 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.255319    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1f2098-72a2-441b-a918-895637f761cf" containerName="revoke-keys"
Oct 31 19:48:40.260316 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.260294    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11068-57371e"
Oct 31 19:48:40.306529 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.306488    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c447j\" (UniqueName: \"kubernetes.io/projected/41987996-a76a-4c00-8d68-6fbf0a632008-kube-api-access-c447j\") pod \"test-curl-11068-57371e\" (UID: \"41987996-a76a-4c00-8d68-6fbf0a632008\") " pod="opendatahub/test-curl-11068-57371e"
Oct 31 19:48:40.310863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.310833    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11068-57371e"]
Oct 31 19:48:40.335913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.335879    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xptzx\""
Oct 31 19:48:40.419780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.408680    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c447j\" (UniqueName: \"kubernetes.io/projected/41987996-a76a-4c00-8d68-6fbf0a632008-kube-api-access-c447j\") pod \"test-curl-11068-57371e\" (UID: \"41987996-a76a-4c00-8d68-6fbf0a632008\") " pod="opendatahub/test-curl-11068-57371e"
Oct 31 19:48:40.591673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.584937    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:40.644718 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.644385    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c447j\" (UniqueName: \"kubernetes.io/projected/41987996-a76a-4c00-8d68-6fbf0a632008-kube-api-access-c447j\") pod \"test-curl-11068-57371e\" (UID: \"41987996-a76a-4c00-8d68-6fbf0a632008\") " pod="opendatahub/test-curl-11068-57371e"
Oct 31 19:48:40.879512 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.879476    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11068-57371e"
Oct 31 19:48:40.961737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:40.961639    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:41.262769 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.261390    5006 generic.go:356] "Generic (PLEG): container finished" podID="9eb48c2a-64c3-493e-bae0-fd287b943cbe" containerID="e946974e60a8acff4e67a7d6f220cc148a5be4a2cb3f2e3cf6539dba43bbf58d" exitCode=0
Oct 31 19:48:41.262769 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.261436    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns" event={"ID":"9eb48c2a-64c3-493e-bae0-fd287b943cbe","Type":"ContainerDied","Data":"e946974e60a8acff4e67a7d6f220cc148a5be4a2cb3f2e3cf6539dba43bbf58d"}
Oct 31 19:48:41.298436 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:41.298347    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41987996_a76a_4c00_8d68_6fbf0a632008.slice/crio-e011daee7624e99d55253b9425f845f2f4700f30d4a854b24841dc3c1c126cb7 WatchSource:0}: Error finding container e011daee7624e99d55253b9425f845f2f4700f30d4a854b24841dc3c1c126cb7: Status 404 returned error can't find the container with id e011daee7624e99d55253b9425f845f2f4700f30d4a854b24841dc3c1c126cb7
Oct 31 19:48:41.300874 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.300847    5006 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:48:41.303271 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.303231    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11068-57371e"]
Oct 31 19:48:41.459511 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.459481    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:41.573996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.573754    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-openshift-ca\") pod \"6489a436-a0d9-43c5-a21d-414f083ee94f\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") "
Oct 31 19:48:41.574179 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.574153    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-openshift-ca" pod "6489a436-a0d9-43c5-a21d-414f083ee94f" (UID: "6489a436-a0d9-43c5-a21d-414f083ee94f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:41.574321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.574303    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6489a436-a0d9-43c5-a21d-414f083ee94f-kube-api-access-p8z28\" (UniqueName: \"kubernetes.io/projected/6489a436-a0d9-43c5-a21d-414f083ee94f-kube-api-access-p8z28\") pod \"6489a436-a0d9-43c5-a21d-414f083ee94f\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") "
Oct 31 19:48:41.574387 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.574376    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-xks-ca\") pod \"6489a436-a0d9-43c5-a21d-414f083ee94f\" (UID: \"6489a436-a0d9-43c5-a21d-414f083ee94f\") "
Oct 31 19:48:41.575142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.575114    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:41.576944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.576915    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-xks-ca" pod "6489a436-a0d9-43c5-a21d-414f083ee94f" (UID: "6489a436-a0d9-43c5-a21d-414f083ee94f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:41.577088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.577071    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6489a436-a0d9-43c5-a21d-414f083ee94f-kube-api-access-p8z28" pod "6489a436-a0d9-43c5-a21d-414f083ee94f" (UID: "6489a436-a0d9-43c5-a21d-414f083ee94f"). InnerVolumeSpecName "kube-api-access-p8z28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:41.676641 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.676603    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8z28\" (UniqueName: \"kubernetes.io/projected/6489a436-a0d9-43c5-a21d-414f083ee94f-kube-api-access-p8z28\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:41.676641 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:41.676636    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6489a436-a0d9-43c5-a21d-414f083ee94f-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:42.272726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.272690    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"
Oct 31 19:48:42.273184 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.272691    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m" event={"ID":"6489a436-a0d9-43c5-a21d-414f083ee94f","Type":"ContainerDied","Data":"0ff9639b124bf53b4701377f5cfc0b2106b2fae7572a7f7606d6e5c4906ff18c"}
Oct 31 19:48:42.273184 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.272868    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ff9639b124bf53b4701377f5cfc0b2106b2fae7572a7f7606d6e5c4906ff18c"
Oct 31 19:48:42.274632 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.274599    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11068-57371e" event={"ID":"41987996-a76a-4c00-8d68-6fbf0a632008","Type":"ContainerStarted","Data":"e011daee7624e99d55253b9425f845f2f4700f30d4a854b24841dc3c1c126cb7"}
Oct 31 19:48:42.493977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.493947    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:42.590944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.590913    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-xks-ca\") pod \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") "
Oct 31 19:48:42.591132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.591060    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb48c2a-64c3-493e-bae0-fd287b943cbe-kube-api-access-jk8wf\" (UniqueName: \"kubernetes.io/projected/9eb48c2a-64c3-493e-bae0-fd287b943cbe-kube-api-access-jk8wf\") pod \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") "
Oct 31 19:48:42.591214 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.591197    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-openshift-ca\") pod \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\" (UID: \"9eb48c2a-64c3-493e-bae0-fd287b943cbe\") "
Oct 31 19:48:42.591570 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.591540    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-openshift-ca" pod "9eb48c2a-64c3-493e-bae0-fd287b943cbe" (UID: "9eb48c2a-64c3-493e-bae0-fd287b943cbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:42.591931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.591911    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:42.593406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.593381    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb48c2a-64c3-493e-bae0-fd287b943cbe-kube-api-access-jk8wf" pod "9eb48c2a-64c3-493e-bae0-fd287b943cbe" (UID: "9eb48c2a-64c3-493e-bae0-fd287b943cbe"). InnerVolumeSpecName "kube-api-access-jk8wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:42.593517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.593491    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-xks-ca" pod "9eb48c2a-64c3-493e-bae0-fd287b943cbe" (UID: "9eb48c2a-64c3-493e-bae0-fd287b943cbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:42.692869 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.692833    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb48c2a-64c3-493e-bae0-fd287b943cbe-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:42.692869 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:42.692866    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk8wf\" (UniqueName: \"kubernetes.io/projected/9eb48c2a-64c3-493e-bae0-fd287b943cbe-kube-api-access-jk8wf\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:43.284756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:43.284702    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns" event={"ID":"9eb48c2a-64c3-493e-bae0-fd287b943cbe","Type":"ContainerDied","Data":"4e6178f7ba037a80b9c19a7929163d96ed3b72f44284a6036b1f429e16e8524b"}
Oct 31 19:48:43.285239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:43.284800    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e6178f7ba037a80b9c19a7929163d96ed3b72f44284a6036b1f429e16e8524b"
Oct 31 19:48:43.285239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:43.284731    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"
Oct 31 19:48:43.286760 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:43.286727    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11068-57371e" event={"ID":"41987996-a76a-4c00-8d68-6fbf0a632008","Type":"ContainerStarted","Data":"6f41c9069071771493256daa6ade9bb7e51fc0b3b33296fb27a9261bfc9c4ee2"}
Oct 31 19:48:43.402079 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:43.402017    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11068-57371e" podStartSLOduration=2.273765489 podStartE2EDuration="3.402001892s" totalImagesPullingTime="1.128236403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:43.398682197 +0000 UTC m=+1941.928216262" watchObservedRunningTime="2026-10-31 19:48:43.402001892 +0000 UTC m=+1941.931535945"
Oct 31 19:48:43.900414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:43.900371    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:44.130806 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:44.130765    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:44.897302 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:44.897259    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11068-57371e"]
Oct 31 19:48:44.905979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:44.905939    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11068-57371e"]
Oct 31 19:48:44.930484 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:44.930447    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:45.305969 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:45.305827    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11068-57371e" podUID="41987996-a76a-4c00-8d68-6fbf0a632008" containerName="test-curl-11068-57371e" containerID="cri-o://6f41c9069071771493256daa6ade9bb7e51fc0b3b33296fb27a9261bfc9c4ee2" gracePeriod=2
Oct 31 19:48:45.357395 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:48:45.357349    5006 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11068-57371e\" is forbidden: User \"system:node:ip-10-0-128-104.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-104.us-west-2.compute.internal' and this object" podUID="41987996-a76a-4c00-8d68-6fbf0a632008" pod="opendatahub/test-curl-11068-57371e"
Oct 31 19:48:45.767584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:45.767547    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:46.404781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:46.374491    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:46.721725 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:46.721585    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:47.193058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.193002    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:47.338746 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.338705    5006 generic.go:356] "Generic (PLEG): container finished" podID="41987996-a76a-4c00-8d68-6fbf0a632008" containerID="6f41c9069071771493256daa6ade9bb7e51fc0b3b33296fb27a9261bfc9c4ee2" exitCode=137
Oct 31 19:48:47.421365 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.421331    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:47.583190 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.583162    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11068-57371e"
Oct 31 19:48:47.691675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.689057    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"]
Oct 31 19:48:47.706780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.704113    5006 cpu_manager.go:414] "containerMap: removing container" podUID="6489a436-a0d9-43c5-a21d-414f083ee94f" containerName="revoke-keys"
Oct 31 19:48:47.706780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.704145    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6489a436-a0d9-43c5-a21d-414f083ee94f" containerName="revoke-keys"
Oct 31 19:48:47.706780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.704196    5006 cpu_manager.go:414] "containerMap: removing container" podUID="41987996-a76a-4c00-8d68-6fbf0a632008" containerName="test-curl-11068-57371e"
Oct 31 19:48:47.706780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.704205    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41987996-a76a-4c00-8d68-6fbf0a632008" containerName="test-curl-11068-57371e"
Oct 31 19:48:47.706780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.704248    5006 cpu_manager.go:414] "containerMap: removing container" podUID="9eb48c2a-64c3-493e-bae0-fd287b943cbe" containerName="revoke-keys"
Oct 31 19:48:47.706780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.704256    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb48c2a-64c3-493e-bae0-fd287b943cbe" containerName="revoke-keys"
Oct 31 19:48:47.706780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.704423    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="41987996-a76a-4c00-8d68-6fbf0a632008" containerName="test-curl-11068-57371e"
Oct 31 19:48:47.706780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.704444    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb48c2a-64c3-493e-bae0-fd287b943cbe" containerName="revoke-keys"
Oct 31 19:48:47.706780 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.704458    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6489a436-a0d9-43c5-a21d-414f083ee94f" containerName="revoke-keys"
Oct 31 19:48:47.720686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.707473    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41987996-a76a-4c00-8d68-6fbf0a632008-kube-api-access-c447j\" (UniqueName: \"kubernetes.io/projected/41987996-a76a-4c00-8d68-6fbf0a632008-kube-api-access-c447j\") pod \"41987996-a76a-4c00-8d68-6fbf0a632008\" (UID: \"41987996-a76a-4c00-8d68-6fbf0a632008\") "
Oct 31 19:48:47.722795 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.722011    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:48:47.763050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.751177    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"]
Oct 31 19:48:47.792624 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.782513    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9be7bd13\""
Oct 31 19:48:47.804722 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.803863    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41987996-a76a-4c00-8d68-6fbf0a632008-kube-api-access-c447j" pod "41987996-a76a-4c00-8d68-6fbf0a632008" (UID: "41987996-a76a-4c00-8d68-6fbf0a632008"). InnerVolumeSpecName "kube-api-access-c447j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:47.811358 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.811127    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzqkj\" (UniqueName: \"kubernetes.io/projected/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-kube-api-access-kzqkj\") pod \"payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h\" (UID: \"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1\") " pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:48:47.811559 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.811399    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-plugins-config-volume\") pod \"payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h\" (UID: \"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1\") " pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:48:47.813233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.813203    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c447j\" (UniqueName: \"kubernetes.io/projected/41987996-a76a-4c00-8d68-6fbf0a632008-kube-api-access-c447j\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:47.833435 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.833395    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9be7bd13-dockercfg-x4t8r\""
Oct 31 19:48:47.838406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.838371    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"]
Oct 31 19:48:47.843165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.843139    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:48:47.895901 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.895868    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"]
Oct 31 19:48:47.915220 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.915180    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3249705a-b2e0-4b8f-8757-661f9b936732-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln\" (UID: \"3249705a-b2e0-4b8f-8757-661f9b936732\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:48:47.915439 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.915344    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzqkj\" (UniqueName: \"kubernetes.io/projected/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-kube-api-access-kzqkj\") pod \"payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h\" (UID: \"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1\") " pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:48:47.915508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.915443    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-plugins-config-volume\") pod \"payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h\" (UID: \"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1\") " pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:48:47.915673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.915629    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88kvt\" (UniqueName: \"kubernetes.io/projected/3249705a-b2e0-4b8f-8757-661f9b936732-kube-api-access-88kvt\") pod \"payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln\" (UID: \"3249705a-b2e0-4b8f-8757-661f9b936732\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:48:47.919987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.917038    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-plugins-config-volume\") pod \"payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h\" (UID: \"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1\") " pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:48:47.976551 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:47.976470    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzqkj\" (UniqueName: \"kubernetes.io/projected/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-kube-api-access-kzqkj\") pod \"payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h\" (UID: \"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1\") " pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:48:48.016969 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.016926    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88kvt\" (UniqueName: \"kubernetes.io/projected/3249705a-b2e0-4b8f-8757-661f9b936732-kube-api-access-88kvt\") pod \"payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln\" (UID: \"3249705a-b2e0-4b8f-8757-661f9b936732\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:48:48.017174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.017048    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3249705a-b2e0-4b8f-8757-661f9b936732-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln\" (UID: \"3249705a-b2e0-4b8f-8757-661f9b936732\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:48:48.018134 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.018106    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3249705a-b2e0-4b8f-8757-661f9b936732-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln\" (UID: \"3249705a-b2e0-4b8f-8757-661f9b936732\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:48:48.074296 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.074265    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88kvt\" (UniqueName: \"kubernetes.io/projected/3249705a-b2e0-4b8f-8757-661f9b936732-kube-api-access-88kvt\") pod \"payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln\" (UID: \"3249705a-b2e0-4b8f-8757-661f9b936732\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:48:48.109385 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.109346    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:48:48.160662 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.160615    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:48:48.379845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.377934    5006 scope.go:122] "RemoveContainer" containerID="6f41c9069071771493256daa6ade9bb7e51fc0b3b33296fb27a9261bfc9c4ee2"
Oct 31 19:48:48.379845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.378158    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11068-57371e"
Oct 31 19:48:48.499839 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:48.499803    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd2e91d_668e_4bf6_9ecd_f95b961a20e1.slice/crio-61a39d3f5c88bc28c157554b1a2b5d68c88fc468ea2bd02da8979278247f92f9 WatchSource:0}: Error finding container 61a39d3f5c88bc28c157554b1a2b5d68c88fc468ea2bd02da8979278247f92f9: Status 404 returned error can't find the container with id 61a39d3f5c88bc28c157554b1a2b5d68c88fc468ea2bd02da8979278247f92f9
Oct 31 19:48:48.505408 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.505382    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"]
Oct 31 19:48:48.577375 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.577287    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:48.593293 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:48.593261    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3249705a_b2e0_4b8f_8757_661f9b936732.slice/crio-4890c0a15261e00c515dc4a1bf55cb5b73f986c5fa9c6c2d7af66e2144569f61 WatchSource:0}: Error finding container 4890c0a15261e00c515dc4a1bf55cb5b73f986c5fa9c6c2d7af66e2144569f61: Status 404 returned error can't find the container with id 4890c0a15261e00c515dc4a1bf55cb5b73f986c5fa9c6c2d7af66e2144569f61
Oct 31 19:48:48.606061 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.603864    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"]
Oct 31 19:48:48.803770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:48.803678    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:49.416677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:49.408593    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h" event={"ID":"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1","Type":"ContainerStarted","Data":"c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7"}
Oct 31 19:48:49.416677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:49.408689    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h" event={"ID":"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1","Type":"ContainerStarted","Data":"61a39d3f5c88bc28c157554b1a2b5d68c88fc468ea2bd02da8979278247f92f9"}
Oct 31 19:48:49.416677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:49.416445    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:48:49.426675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:49.423241    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln" event={"ID":"3249705a-b2e0-4b8f-8757-661f9b936732","Type":"ContainerStarted","Data":"96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0"}
Oct 31 19:48:49.426675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:49.423310    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln" event={"ID":"3249705a-b2e0-4b8f-8757-661f9b936732","Type":"ContainerStarted","Data":"4890c0a15261e00c515dc4a1bf55cb5b73f986c5fa9c6c2d7af66e2144569f61"}
Oct 31 19:48:49.426675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:49.423577    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:48:49.538577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:49.538521    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h" podStartSLOduration=2.538507138 podStartE2EDuration="2.538507138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:49.537311065 +0000 UTC m=+1948.066845138" watchObservedRunningTime="2026-10-31 19:48:49.538507138 +0000 UTC m=+1948.068041191"
Oct 31 19:48:49.604377 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:49.604340    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41987996-a76a-4c00-8d68-6fbf0a632008" path="/var/lib/kubelet/pods/41987996-a76a-4c00-8d68-6fbf0a632008/volumes"
Oct 31 19:48:49.656498 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:49.656437    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln" podStartSLOduration=2.656420942 podStartE2EDuration="2.656420942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:49.65180827 +0000 UTC m=+1948.181342322" watchObservedRunningTime="2026-10-31 19:48:49.656420942 +0000 UTC m=+1948.185954994"
Oct 31 19:48:49.949472 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:49.949433    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:50.182102 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:50.182071    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:50.412265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:50.412228    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:50.662187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:50.662146    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:50.902444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:50.902404    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:53.755038 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:53.755003    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:53.975201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:53.975162    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:54.583464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:54.583426    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:54.826524 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:54.826484    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:54.979815 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:54.979779    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"]
Oct 31 19:48:54.985913 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:54.985888    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:48:55.037050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.037015    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6kkf\" (UniqueName: \"kubernetes.io/projected/6141a7b4-2875-4dad-90b8-2576b934b347-kube-api-access-n6kkf\") pod \"payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd\" (UID: \"6141a7b4-2875-4dad-90b8-2576b934b347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:48:55.037262 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.037071    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6141a7b4-2875-4dad-90b8-2576b934b347-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd\" (UID: \"6141a7b4-2875-4dad-90b8-2576b934b347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:48:55.048406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.048367    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"]
Oct 31 19:48:55.093462 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.093420    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ae778663\""
Oct 31 19:48:55.093730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.093469    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ae778663-dockercfg-l848s\""
Oct 31 19:48:55.138385 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.138335    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6141a7b4-2875-4dad-90b8-2576b934b347-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd\" (UID: \"6141a7b4-2875-4dad-90b8-2576b934b347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:48:55.138709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.138688    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6kkf\" (UniqueName: \"kubernetes.io/projected/6141a7b4-2875-4dad-90b8-2576b934b347-kube-api-access-n6kkf\") pod \"payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd\" (UID: \"6141a7b4-2875-4dad-90b8-2576b934b347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:48:55.139169 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.139144    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6141a7b4-2875-4dad-90b8-2576b934b347-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd\" (UID: \"6141a7b4-2875-4dad-90b8-2576b934b347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:48:55.145603 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.145556    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:55.214805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.214770    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6kkf\" (UniqueName: \"kubernetes.io/projected/6141a7b4-2875-4dad-90b8-2576b934b347-kube-api-access-n6kkf\") pod \"payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd\" (UID: \"6141a7b4-2875-4dad-90b8-2576b934b347\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:48:55.301720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.301565    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:48:55.388156 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.388126    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:55.702375 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:55.702336    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6141a7b4_2875_4dad_90b8_2576b934b347.slice/crio-9f0797bf8fe46cea89e91ce8843ec51ff8cb1101e341656fba7f6c6315d3b567 WatchSource:0}: Error finding container 9f0797bf8fe46cea89e91ce8843ec51ff8cb1101e341656fba7f6c6315d3b567: Status 404 returned error can't find the container with id 9f0797bf8fe46cea89e91ce8843ec51ff8cb1101e341656fba7f6c6315d3b567
Oct 31 19:48:55.708635 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:55.708603    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"]
Oct 31 19:48:56.449871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:56.449832    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:56.496315 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:56.496275    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd" event={"ID":"6141a7b4-2875-4dad-90b8-2576b934b347","Type":"ContainerStarted","Data":"a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290"}
Oct 31 19:48:56.496538 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:56.496421    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:48:56.496538 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:56.496472    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd" event={"ID":"6141a7b4-2875-4dad-90b8-2576b934b347","Type":"ContainerStarted","Data":"9f0797bf8fe46cea89e91ce8843ec51ff8cb1101e341656fba7f6c6315d3b567"}
Oct 31 19:48:56.613923 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:56.613870    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd" podStartSLOduration=2.613851288 podStartE2EDuration="2.613851288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:56.610536407 +0000 UTC m=+1955.140070460" watchObservedRunningTime="2026-10-31 19:48:56.613851288 +0000 UTC m=+1955.143385352"
Oct 31 19:48:57.373304 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.373268    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"]
Oct 31 19:48:57.377859 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.377817    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:57.433125 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.433073    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"]
Oct 31 19:48:57.476709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.476163    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:48:57.476709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.476266    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:48:57.476709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.476324    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:48:57.481121 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.481093    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:48:57.490332 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.490285    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12d17959-c6g2r\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:57.490505 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.490483    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj262\" (UniqueName: \"kubernetes.io/projected/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-kube-api-access-lj262\") pod \"maas-api-revoke-sub-12d17959-c6g2r\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:57.490662 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.490627    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12d17959-c6g2r\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:57.592050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.592002    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12d17959-c6g2r\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:57.592259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.592094    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj262\" (UniqueName: \"kubernetes.io/projected/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-kube-api-access-lj262\") pod \"maas-api-revoke-sub-12d17959-c6g2r\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:57.592259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.592153    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12d17959-c6g2r\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:57.592856 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.592827    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12d17959-c6g2r\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:57.594712 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.594685    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12d17959-c6g2r\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:57.649887 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.649814    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj262\" (UniqueName: \"kubernetes.io/projected/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-kube-api-access-lj262\") pod \"maas-api-revoke-sub-12d17959-c6g2r\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:57.698814 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:57.698779    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:48:58.074258 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:48:58.074220    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89545a2_6e00_4f0e_bc41_5e6733ad9b93.slice/crio-9fd6f72341876c6303bf8690129f441449af097894e5336d3fc49faf1bfcd994 WatchSource:0}: Error finding container 9fd6f72341876c6303bf8690129f441449af097894e5336d3fc49faf1bfcd994: Status 404 returned error can't find the container with id 9fd6f72341876c6303bf8690129f441449af097894e5336d3fc49faf1bfcd994
Oct 31 19:48:58.079841 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:58.079814    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"]
Oct 31 19:48:58.515303 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:58.515261    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r" event={"ID":"c89545a2-6e00-4f0e-bc41-5e6733ad9b93","Type":"ContainerStarted","Data":"75199fde0794622fe504ac5d313aabc01f6097f5a5efa4b08584c95ba0f81d7b"}
Oct 31 19:48:58.515840 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:58.515333    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r" event={"ID":"c89545a2-6e00-4f0e-bc41-5e6733ad9b93","Type":"ContainerStarted","Data":"9fd6f72341876c6303bf8690129f441449af097894e5336d3fc49faf1bfcd994"}
Oct 31 19:48:58.630265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:58.630206    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r" podStartSLOduration=1.6301907340000001 podStartE2EDuration="1.630190734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:58.62874447 +0000 UTC m=+1957.158278522" watchObservedRunningTime="2026-10-31 19:48:58.630190734 +0000 UTC m=+1957.159724787"
Oct 31 19:48:59.525314 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:59.525276    5006 generic.go:356] "Generic (PLEG): container finished" podID="c89545a2-6e00-4f0e-bc41-5e6733ad9b93" containerID="75199fde0794622fe504ac5d313aabc01f6097f5a5efa4b08584c95ba0f81d7b" exitCode=0
Oct 31 19:48:59.525793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:59.525361    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r" event={"ID":"c89545a2-6e00-4f0e-bc41-5e6733ad9b93","Type":"ContainerDied","Data":"75199fde0794622fe504ac5d313aabc01f6097f5a5efa4b08584c95ba0f81d7b"}
Oct 31 19:48:59.713215 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:59.713175    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:59.935580 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:48:59.935542    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:00.041166 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.041129    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"]
Oct 31 19:49:00.046724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.046696    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:49:00.104389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.104351    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"]
Oct 31 19:49:00.136973 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.136934    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vstkm\" (UniqueName: \"kubernetes.io/projected/a429faf1-b779-4be2-a8a2-ad6a71558cc0-kube-api-access-vstkm\") pod \"payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8\" (UID: \"a429faf1-b779-4be2-a8a2-ad6a71558cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:49:00.137517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.137490    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a429faf1-b779-4be2-a8a2-ad6a71558cc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8\" (UID: \"a429faf1-b779-4be2-a8a2-ad6a71558cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:49:00.144368 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.144323    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-31a1ba53-dockercfg-tftht\""
Oct 31 19:49:00.145451 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.145422    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-31a1ba53\""
Oct 31 19:49:00.242910 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.242824    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstkm\" (UniqueName: \"kubernetes.io/projected/a429faf1-b779-4be2-a8a2-ad6a71558cc0-kube-api-access-vstkm\") pod \"payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8\" (UID: \"a429faf1-b779-4be2-a8a2-ad6a71558cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:49:00.243080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.243041    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a429faf1-b779-4be2-a8a2-ad6a71558cc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8\" (UID: \"a429faf1-b779-4be2-a8a2-ad6a71558cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:49:00.244479 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.244430    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a429faf1-b779-4be2-a8a2-ad6a71558cc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8\" (UID: \"a429faf1-b779-4be2-a8a2-ad6a71558cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:49:00.303826 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.303790    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstkm\" (UniqueName: \"kubernetes.io/projected/a429faf1-b779-4be2-a8a2-ad6a71558cc0-kube-api-access-vstkm\") pod \"payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8\" (UID: \"a429faf1-b779-4be2-a8a2-ad6a71558cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:49:00.364922 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.364891    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:49:00.442901 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.437742    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:49:00.760795 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.760764    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"]
Oct 31 19:49:00.786270 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.786242    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:49:00.854261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.854230    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-xks-ca\") pod \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") "
Oct 31 19:49:00.854445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.854430    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-openshift-ca\") pod \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") "
Oct 31 19:49:00.854502 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.854490    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-kube-api-access-lj262\" (UniqueName: \"kubernetes.io/projected/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-kube-api-access-lj262\") pod \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\" (UID: \"c89545a2-6e00-4f0e-bc41-5e6733ad9b93\") "
Oct 31 19:49:00.854842 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.854813    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-openshift-ca" pod "c89545a2-6e00-4f0e-bc41-5e6733ad9b93" (UID: "c89545a2-6e00-4f0e-bc41-5e6733ad9b93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:00.855040 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.855022    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:00.896184 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.896144    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-xks-ca" pod "c89545a2-6e00-4f0e-bc41-5e6733ad9b93" (UID: "c89545a2-6e00-4f0e-bc41-5e6733ad9b93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:00.896367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.896252    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-kube-api-access-lj262" pod "c89545a2-6e00-4f0e-bc41-5e6733ad9b93" (UID: "c89545a2-6e00-4f0e-bc41-5e6733ad9b93"). InnerVolumeSpecName "kube-api-access-lj262". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:00.956398 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.956361    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj262\" (UniqueName: \"kubernetes.io/projected/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-kube-api-access-lj262\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:00.956398 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:00.956392    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89545a2-6e00-4f0e-bc41-5e6733ad9b93-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:01.077947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:01.077814    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:01.448893 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:01.448863    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:49:01.552807 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:01.552771    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8" event={"ID":"a429faf1-b779-4be2-a8a2-ad6a71558cc0","Type":"ContainerStarted","Data":"b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0"}
Oct 31 19:49:01.552991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:01.552833    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8" event={"ID":"a429faf1-b779-4be2-a8a2-ad6a71558cc0","Type":"ContainerStarted","Data":"97c2d7e18a96ec03fa257b3064b01136252f9e86a73c0c6983d49ba0f77fbc49"}
Oct 31 19:49:01.552991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:01.552876    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:49:01.554378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:01.554356    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"
Oct 31 19:49:01.554550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:01.554394    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r" event={"ID":"c89545a2-6e00-4f0e-bc41-5e6733ad9b93","Type":"ContainerDied","Data":"9fd6f72341876c6303bf8690129f441449af097894e5336d3fc49faf1bfcd994"}
Oct 31 19:49:01.554550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:01.554438    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fd6f72341876c6303bf8690129f441449af097894e5336d3fc49faf1bfcd994"
Oct 31 19:49:01.683285 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:01.683223    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8" podStartSLOduration=2.683207107 podStartE2EDuration="2.683207107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:01.681388115 +0000 UTC m=+1960.210922168" watchObservedRunningTime="2026-10-31 19:49:01.683207107 +0000 UTC m=+1960.212741160"
Oct 31 19:49:01.805578 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:01.805475    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:05.396640 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:49:05.396595    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:49:07.508128 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:07.508089    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:07.508873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:07.508854    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:49:08.213971 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:08.213934    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:09.071820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.071766    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"]
Oct 31 19:49:09.072504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.072484    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c89545a2-6e00-4f0e-bc41-5e6733ad9b93" containerName="revoke-keys"
Oct 31 19:49:09.072504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.072503    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89545a2-6e00-4f0e-bc41-5e6733ad9b93" containerName="revoke-keys"
Oct 31 19:49:09.072690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.072638    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89545a2-6e00-4f0e-bc41-5e6733ad9b93" containerName="revoke-keys"
Oct 31 19:49:09.078210 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.078180    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:09.129664 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.129616    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"]
Oct 31 19:49:09.173516 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.173480    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:49:09.173762 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.173546    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:49:09.173762 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.173546    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:49:09.181308 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.181268    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:49:09.243002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.242962    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtlk5\" (UniqueName: \"kubernetes.io/projected/d1cb72c2-4899-4301-b457-c594a5f52437-kube-api-access-qtlk5\") pod \"maas-api-revoke-sub-0d4f5dc5-c6l7f\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:09.243205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.243186    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d4f5dc5-c6l7f\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:09.243271 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.243237    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d4f5dc5-c6l7f\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:09.344477 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.344432    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d4f5dc5-c6l7f\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:09.344709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.344518    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d4f5dc5-c6l7f\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:09.344709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.344695    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtlk5\" (UniqueName: \"kubernetes.io/projected/d1cb72c2-4899-4301-b457-c594a5f52437-kube-api-access-qtlk5\") pod \"maas-api-revoke-sub-0d4f5dc5-c6l7f\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:09.345170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.345143    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d4f5dc5-c6l7f\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:09.347198 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.347168    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d4f5dc5-c6l7f\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:09.403486 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.403451    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtlk5\" (UniqueName: \"kubernetes.io/projected/d1cb72c2-4899-4301-b457-c594a5f52437-kube-api-access-qtlk5\") pod \"maas-api-revoke-sub-0d4f5dc5-c6l7f\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:09.697079 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:09.696982    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:10.089961 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:49:10.089925    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cb72c2_4899_4301_b457_c594a5f52437.slice/crio-aece01df252d2e8d28a43217b1b4bb0779100f8e2edb17d93a6c3b9533015cda WatchSource:0}: Error finding container aece01df252d2e8d28a43217b1b4bb0779100f8e2edb17d93a6c3b9533015cda: Status 404 returned error can't find the container with id aece01df252d2e8d28a43217b1b4bb0779100f8e2edb17d93a6c3b9533015cda
Oct 31 19:49:10.094944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:10.094912    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"]
Oct 31 19:49:10.718202 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:10.718158    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f" event={"ID":"d1cb72c2-4899-4301-b457-c594a5f52437","Type":"ContainerStarted","Data":"ff372ccd1aaf1272c7b120cdf22ff78d1226aad90f9971fa142eb39118298fd0"}
Oct 31 19:49:10.718414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:10.718228    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f" event={"ID":"d1cb72c2-4899-4301-b457-c594a5f52437","Type":"ContainerStarted","Data":"aece01df252d2e8d28a43217b1b4bb0779100f8e2edb17d93a6c3b9533015cda"}
Oct 31 19:49:10.832837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:10.832731    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f" podStartSLOduration=1.832717013 podStartE2EDuration="1.832717013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:10.831339708 +0000 UTC m=+1969.360873760" watchObservedRunningTime="2026-10-31 19:49:10.832717013 +0000 UTC m=+1969.362251066"
Oct 31 19:49:11.054034 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.053990    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"]
Oct 31 19:49:11.058800 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.058771    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.112223 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.112184    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 19:49:11.112223 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.112223    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hnzhb\""
Oct 31 19:49:11.112742 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.112244    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 19:49:11.112813 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.112206    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 19:49:11.115224 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.115195    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"]
Oct 31 19:49:11.174049 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.173922    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.174373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.174348    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.174698 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.174426    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.174698 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.174534    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/476b810f-d23d-475f-be1c-2d08b360a899-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.174698 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.174607    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.174905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.174800    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7jbk\" (UniqueName: \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-kube-api-access-j7jbk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.174905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.174837    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.276274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.276236    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7jbk\" (UniqueName: \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-kube-api-access-j7jbk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.276274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.276287    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.276565 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.276348    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.276565 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.276386    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.276565 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.276446    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.276565 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.276545    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/476b810f-d23d-475f-be1c-2d08b360a899-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.276832 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.276635    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.277123 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.277092    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.279063 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.279041    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.279175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.279084    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/476b810f-d23d-475f-be1c-2d08b360a899-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.279175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.279119    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.279301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.279280    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.336181 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.336148    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.336181 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.336170    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7jbk\" (UniqueName: \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-kube-api-access-j7jbk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xg4nr\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.448244 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.448143    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:11.730835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.730749    5006 generic.go:356] "Generic (PLEG): container finished" podID="d1cb72c2-4899-4301-b457-c594a5f52437" containerID="ff372ccd1aaf1272c7b120cdf22ff78d1226aad90f9971fa142eb39118298fd0" exitCode=0
Oct 31 19:49:11.730991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.730826    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f" event={"ID":"d1cb72c2-4899-4301-b457-c594a5f52437","Type":"ContainerDied","Data":"ff372ccd1aaf1272c7b120cdf22ff78d1226aad90f9971fa142eb39118298fd0"}
Oct 31 19:49:11.910936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:11.910644    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"]
Oct 31 19:49:11.918953 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:49:11.918923    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476b810f_d23d_475f_be1c_2d08b360a899.slice/crio-a061628bf44717715123fd5d254fe03c8a294bb44f0845581d56c6b0037a15f4 WatchSource:0}: Error finding container a061628bf44717715123fd5d254fe03c8a294bb44f0845581d56c6b0037a15f4: Status 404 returned error can't find the container with id a061628bf44717715123fd5d254fe03c8a294bb44f0845581d56c6b0037a15f4
Oct 31 19:49:12.386788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:12.386754    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:12.567038 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:12.567010    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:49:12.744985 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:12.744896    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr" event={"ID":"476b810f-d23d-475f-be1c-2d08b360a899","Type":"ContainerStarted","Data":"a061628bf44717715123fd5d254fe03c8a294bb44f0845581d56c6b0037a15f4"}
Oct 31 19:49:12.944172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:12.944140    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:13.104392 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.104346    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cb72c2-4899-4301-b457-c594a5f52437-kube-api-access-qtlk5\" (UniqueName: \"kubernetes.io/projected/d1cb72c2-4899-4301-b457-c594a5f52437-kube-api-access-qtlk5\") pod \"d1cb72c2-4899-4301-b457-c594a5f52437\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") "
Oct 31 19:49:13.104584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.104483    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-openshift-ca\") pod \"d1cb72c2-4899-4301-b457-c594a5f52437\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") "
Oct 31 19:49:13.104584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.104546    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-xks-ca\") pod \"d1cb72c2-4899-4301-b457-c594a5f52437\" (UID: \"d1cb72c2-4899-4301-b457-c594a5f52437\") "
Oct 31 19:49:13.105015 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.104982    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-openshift-ca" pod "d1cb72c2-4899-4301-b457-c594a5f52437" (UID: "d1cb72c2-4899-4301-b457-c594a5f52437"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:13.105453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.105425    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:13.108359 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.108325    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cb72c2-4899-4301-b457-c594a5f52437-kube-api-access-qtlk5" pod "d1cb72c2-4899-4301-b457-c594a5f52437" (UID: "d1cb72c2-4899-4301-b457-c594a5f52437"). InnerVolumeSpecName "kube-api-access-qtlk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:13.108494 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.108433    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-xks-ca" pod "d1cb72c2-4899-4301-b457-c594a5f52437" (UID: "d1cb72c2-4899-4301-b457-c594a5f52437"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:13.207406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.207369    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtlk5\" (UniqueName: \"kubernetes.io/projected/d1cb72c2-4899-4301-b457-c594a5f52437-kube-api-access-qtlk5\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:13.207406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.207398    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cb72c2-4899-4301-b457-c594a5f52437-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:13.755422 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.755387    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"
Oct 31 19:49:13.755883 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.755391    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f" event={"ID":"d1cb72c2-4899-4301-b457-c594a5f52437","Type":"ContainerDied","Data":"aece01df252d2e8d28a43217b1b4bb0779100f8e2edb17d93a6c3b9533015cda"}
Oct 31 19:49:13.755883 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:13.755513    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aece01df252d2e8d28a43217b1b4bb0779100f8e2edb17d93a6c3b9533015cda"
Oct 31 19:49:14.220840 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:14.220791    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:16.740492 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:16.740452    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:16.740873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:16.740527    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:17.275911 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:17.275873    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:17.800143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:17.800096    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr" event={"ID":"476b810f-d23d-475f-be1c-2d08b360a899","Type":"ContainerStarted","Data":"1d60b1cc0d098a3a8b0387fb10367f9f3f3d50a7ae7705ab64c052fdcd5f62e5"}
Oct 31 19:49:17.800631 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:17.800343    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:17.803131 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:17.803095    5006 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xg4nr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:49:17.803276 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:17.803160    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr" podUID="476b810f-d23d-475f-be1c-2d08b360a899" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:49:18.014413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:18.014345    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr" podStartSLOduration=2.19685874 podStartE2EDuration="7.014327587s" totalImagesPullingTime="4.817468847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:17.969170148 +0000 UTC m=+1976.498704203" watchObservedRunningTime="2026-10-31 19:49:18.014327587 +0000 UTC m=+1976.543861637"
Oct 31 19:49:18.812050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:18.812019    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 19:49:19.830273 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.830235    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"]
Oct 31 19:49:19.831019 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.831003    5006 cpu_manager.go:414] "containerMap: removing container" podUID="d1cb72c2-4899-4301-b457-c594a5f52437" containerName="revoke-keys"
Oct 31 19:49:19.831095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.831021    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cb72c2-4899-4301-b457-c594a5f52437" containerName="revoke-keys"
Oct 31 19:49:19.831164 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.831155    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cb72c2-4899-4301-b457-c594a5f52437" containerName="revoke-keys"
Oct 31 19:49:19.835208 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.835188    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:19.887933 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.887898    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"]
Oct 31 19:49:19.905565 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.905523    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87cc9a2-nt5l5\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:19.906038 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.906010    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgfds\" (UniqueName: \"kubernetes.io/projected/59d0fd17-18c7-4c99-9731-120dfee085d7-kube-api-access-fgfds\") pod \"maas-api-revoke-sub-c87cc9a2-nt5l5\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:19.906163 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.906148    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87cc9a2-nt5l5\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:19.932583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.932536    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:49:19.932583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.932541    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:49:19.933003 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.932978    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:49:19.938143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:19.938116    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:49:20.008274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:20.008228    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87cc9a2-nt5l5\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:20.008459 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:20.008394    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgfds\" (UniqueName: \"kubernetes.io/projected/59d0fd17-18c7-4c99-9731-120dfee085d7-kube-api-access-fgfds\") pod \"maas-api-revoke-sub-c87cc9a2-nt5l5\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:20.008459 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:20.008420    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87cc9a2-nt5l5\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:20.009170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:20.009143    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87cc9a2-nt5l5\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:20.010790 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:20.010770    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87cc9a2-nt5l5\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:20.066051 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:20.066019    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgfds\" (UniqueName: \"kubernetes.io/projected/59d0fd17-18c7-4c99-9731-120dfee085d7-kube-api-access-fgfds\") pod \"maas-api-revoke-sub-c87cc9a2-nt5l5\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:20.149645 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:20.149607    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:20.529230 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:49:20.528483    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d0fd17_18c7_4c99_9731_120dfee085d7.slice/crio-4a33af36d3c8d268a35544652747242a06bb04cb0ee0df776af511acd24eb459 WatchSource:0}: Error finding container 4a33af36d3c8d268a35544652747242a06bb04cb0ee0df776af511acd24eb459: Status 404 returned error can't find the container with id 4a33af36d3c8d268a35544652747242a06bb04cb0ee0df776af511acd24eb459
Oct 31 19:49:20.533497 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:20.533470    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"]
Oct 31 19:49:20.832318 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:20.832230    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5" event={"ID":"59d0fd17-18c7-4c99-9731-120dfee085d7","Type":"ContainerStarted","Data":"4a33af36d3c8d268a35544652747242a06bb04cb0ee0df776af511acd24eb459"}
Oct 31 19:49:21.843533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:21.843495    5006 generic.go:356] "Generic (PLEG): container finished" podID="59d0fd17-18c7-4c99-9731-120dfee085d7" containerID="504d14cfff821e7786fcee6e2ca53da99b7157052f9afbc9e96aafb2fdd9e74f" exitCode=0
Oct 31 19:49:21.843956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:21.843584    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5" event={"ID":"59d0fd17-18c7-4c99-9731-120dfee085d7","Type":"ContainerDied","Data":"504d14cfff821e7786fcee6e2ca53da99b7157052f9afbc9e96aafb2fdd9e74f"}
Oct 31 19:49:22.176943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.176848    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:22.180214 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.180189    5006 scope.go:122] "RemoveContainer" containerID="20303049ddfc157f8af2565e0d093ace991fcff4a61ef0103e4be7129fced1cd"
Oct 31 19:49:22.195728 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.195702    5006 scope.go:122] "RemoveContainer" containerID="e8b63fce1f8488bd4c5ccf45cd4542eafb2ccb73a6cda23ecbf5fb3f22f8a7b7"
Oct 31 19:49:22.209507 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.209485    5006 scope.go:122] "RemoveContainer" containerID="d76d80456008111cd870f42c2ca84b49ce9cb2177c3693e1d692548f71bdd459"
Oct 31 19:49:22.221843 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.221804    5006 scope.go:122] "RemoveContainer" containerID="bd04325f20effc08c9b554afac01e6b51b618b4988d001ff682d28254a3ce577"
Oct 31 19:49:22.235593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.235571    5006 scope.go:122] "RemoveContainer" containerID="2dfa08e8f7364b8dbd09dc70662dd7aa2486133e7a9a73c982670ec8ad35fc21"
Oct 31 19:49:22.248735 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.248703    5006 scope.go:122] "RemoveContainer" containerID="8c1c6aa45b70f4efdf251dade736f5c995cf91595f0646416b46e6ea52410d98"
Oct 31 19:49:22.260503 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.260474    5006 scope.go:122] "RemoveContainer" containerID="01fb87885fb6804e8f5e0b19c0e8830a24761f780e6ac74939d8778e36b33152"
Oct 31 19:49:22.272858 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.272830    5006 scope.go:122] "RemoveContainer" containerID="3d78396a9487d1fa96008ab05627e06bedf48e72b26d7f8abe28f8f107c027d2"
Oct 31 19:49:22.284078 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.284052    5006 scope.go:122] "RemoveContainer" containerID="0003ca0182ea9966f51975935f47bf9dfe6910e01b65d6190121fe51a81b0089"
Oct 31 19:49:22.295839 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.295815    5006 scope.go:122] "RemoveContainer" containerID="4ecefcab230e1ac623d098ae46748c22a563108eadcf50f6094fde143aaf8620"
Oct 31 19:49:22.310277 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.310140    5006 scope.go:122] "RemoveContainer" containerID="4e8af9200ad772cc25d27fff3661e73c2134924caa9cb45dfc5d8af764822a3d"
Oct 31 19:49:22.322572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.322545    5006 scope.go:122] "RemoveContainer" containerID="9cb8f72a0c7a16cceb170c538ff5ae7966007717529b4305a0d91032153089de"
Oct 31 19:49:22.334012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.333982    5006 scope.go:122] "RemoveContainer" containerID="df2e4bb5edb981b9c99cda24beb6a01f8dc97545a4fd3c828bbc0ad7647b391c"
Oct 31 19:49:22.345985 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.345963    5006 scope.go:122] "RemoveContainer" containerID="684abb653c6ac2dc973c81e4cf72abc94fc7e3ce59fa43e7de8e5abbe1f00d9d"
Oct 31 19:49:22.361904 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.361878    5006 scope.go:122] "RemoveContainer" containerID="f92695fb1d6e04f5d36c370bb20f13db67988d114079fc2deae9cfe9941437ff"
Oct 31 19:49:22.375207 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.375183    5006 scope.go:122] "RemoveContainer" containerID="0de906dbe60c7f429196853e72a4de06f2924b9f32345ffd4e4318e9d5bcb682"
Oct 31 19:49:22.784930 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:22.784817    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:23.035426 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.035356    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:23.161575 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.161539    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-openshift-ca\") pod \"59d0fd17-18c7-4c99-9731-120dfee085d7\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") "
Oct 31 19:49:23.161788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.161599    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d0fd17-18c7-4c99-9731-120dfee085d7-kube-api-access-fgfds\" (UniqueName: \"kubernetes.io/projected/59d0fd17-18c7-4c99-9731-120dfee085d7-kube-api-access-fgfds\") pod \"59d0fd17-18c7-4c99-9731-120dfee085d7\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") "
Oct 31 19:49:23.161861 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.161804    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-xks-ca\") pod \"59d0fd17-18c7-4c99-9731-120dfee085d7\" (UID: \"59d0fd17-18c7-4c99-9731-120dfee085d7\") "
Oct 31 19:49:23.162032 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.162000    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-openshift-ca" pod "59d0fd17-18c7-4c99-9731-120dfee085d7" (UID: "59d0fd17-18c7-4c99-9731-120dfee085d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:23.162474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.162443    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:23.164169 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.164141    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-xks-ca" pod "59d0fd17-18c7-4c99-9731-120dfee085d7" (UID: "59d0fd17-18c7-4c99-9731-120dfee085d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:23.164628 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.164607    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d0fd17-18c7-4c99-9731-120dfee085d7-kube-api-access-fgfds" pod "59d0fd17-18c7-4c99-9731-120dfee085d7" (UID: "59d0fd17-18c7-4c99-9731-120dfee085d7"). InnerVolumeSpecName "kube-api-access-fgfds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:23.263865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.263824    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgfds\" (UniqueName: \"kubernetes.io/projected/59d0fd17-18c7-4c99-9731-120dfee085d7-kube-api-access-fgfds\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:23.263865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.263857    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d0fd17-18c7-4c99-9731-120dfee085d7-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:23.861502 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.861422    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"
Oct 31 19:49:23.861502 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.861468    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5" event={"ID":"59d0fd17-18c7-4c99-9731-120dfee085d7","Type":"ContainerDied","Data":"4a33af36d3c8d268a35544652747242a06bb04cb0ee0df776af511acd24eb459"}
Oct 31 19:49:23.861751 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:23.861517    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a33af36d3c8d268a35544652747242a06bb04cb0ee0df776af511acd24eb459"
Oct 31 19:49:26.921393 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:26.921347    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:27.138821 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:27.138786    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:27.349739 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:27.349696    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:32.181172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:32.181120    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:33.646508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:33.646472    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:33.890891 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:33.890856    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:34.116349 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.116305    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:34.142000 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.141963    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"]
Oct 31 19:49:34.142720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.142701    5006 cpu_manager.go:414] "containerMap: removing container" podUID="59d0fd17-18c7-4c99-9731-120dfee085d7" containerName="revoke-keys"
Oct 31 19:49:34.142720 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.142720    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59d0fd17-18c7-4c99-9731-120dfee085d7" containerName="revoke-keys"
Oct 31 19:49:34.142870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.142861    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="59d0fd17-18c7-4c99-9731-120dfee085d7" containerName="revoke-keys"
Oct 31 19:49:34.153452 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.153421    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.205922 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.205882    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5cqm6\""
Oct 31 19:49:34.229249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.229204    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.229459 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.229368    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.229459 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.229412    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.229670 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.229632    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.229770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.229746    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7dg\" (UniqueName: \"kubernetes.io/projected/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-kube-api-access-hn7dg\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.229824 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.229807    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.229932 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.229917    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.229971 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.229956    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.230085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.230068    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.332259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.332226    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.332438 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.332289    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.332438 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.332349    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.332438 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.332411    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.332573 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.332436    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.332632 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.332601    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.332753 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.332641    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.332753 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.332673    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7dg\" (UniqueName: \"kubernetes.io/projected/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-kube-api-access-hn7dg\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.332753 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.332738    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.332924 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.332812    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.333111 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.333083    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.333242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.333188    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.333322 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.333260    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.333385 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.333364    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.334986 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.334962    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.335465 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.335447    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.346964 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.346922    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"]
Oct 31 19:49:34.390262 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.390182    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7dg\" (UniqueName: \"kubernetes.io/projected/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-kube-api-access-hn7dg\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.390429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.390336    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14112fb-b3f3-4ba3-8e12-af608d6c6beb-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-cccdj\" (UID: \"b14112fb-b3f3-4ba3-8e12-af608d6c6beb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.470736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.470697    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:34.900886 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.900838    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"]
Oct 31 19:49:34.929570 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:49:34.929525    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14112fb_b3f3_4ba3_8e12_af608d6c6beb.slice/crio-7870416d0d0c2209e5775b9af3e663965a462475091fad6095eb55cde03b6b7c WatchSource:0}: Error finding container 7870416d0d0c2209e5775b9af3e663965a462475091fad6095eb55cde03b6b7c: Status 404 returned error can't find the container with id 7870416d0d0c2209e5775b9af3e663965a462475091fad6095eb55cde03b6b7c
Oct 31 19:49:34.932274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.932240    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:34.932352 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.932321    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:34.932389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.932352    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:34.981731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:34.981699    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" event={"ID":"b14112fb-b3f3-4ba3-8e12-af608d6c6beb","Type":"ContainerStarted","Data":"7870416d0d0c2209e5775b9af3e663965a462475091fad6095eb55cde03b6b7c"}
Oct 31 19:49:35.407096 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:49:35.407054    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:49:35.994148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:35.994098    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" event={"ID":"b14112fb-b3f3-4ba3-8e12-af608d6c6beb","Type":"ContainerStarted","Data":"c88ec4d1cc9e806615d55f605cc419023cb378e365cc8c952561572ab99a98d6"}
Oct 31 19:49:36.147290 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:36.147223    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podStartSLOduration=2.147206558 podStartE2EDuration="2.147206558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:36.143489605 +0000 UTC m=+1994.673023655" watchObservedRunningTime="2026-10-31 19:49:36.147206558 +0000 UTC m=+1994.676740610"
Oct 31 19:49:36.471257 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:36.471216    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:36.473373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:36.473342    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cccdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:49:36.473539 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:36.473397    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podUID="b14112fb-b3f3-4ba3-8e12-af608d6c6beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 19:49:37.033174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:37.032426    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:37.310330 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:37.310235    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:37.471826 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:37.471793    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cccdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:49:37.472011 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:37.471852    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podUID="b14112fb-b3f3-4ba3-8e12-af608d6c6beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 19:49:37.525669 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:37.525621    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:38.471428 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:38.471387    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cccdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:49:38.471962 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:38.471458    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podUID="b14112fb-b3f3-4ba3-8e12-af608d6c6beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 19:49:39.471603 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:39.471565    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cccdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:49:39.472047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:39.471621    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podUID="b14112fb-b3f3-4ba3-8e12-af608d6c6beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 19:49:40.031251 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:40.031214    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:40.265951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:40.265918    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:40.471790 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:40.471753    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cccdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:49:40.472321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:40.471809    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podUID="b14112fb-b3f3-4ba3-8e12-af608d6c6beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 19:49:40.619345 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:40.619307    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:41.472112 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:41.472078    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cccdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:49:41.472504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:41.472130    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podUID="b14112fb-b3f3-4ba3-8e12-af608d6c6beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 19:49:42.308914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:42.308871    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:42.472104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:42.472068    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cccdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 19:49:42.472319 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:42.472165    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podUID="b14112fb-b3f3-4ba3-8e12-af608d6c6beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 19:49:43.640042 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:43.640000    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:43.641543 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:43.641505    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:43.642087 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:43.642064    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj"
Oct 31 19:49:44.339525 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:44.339486    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:49:46.844740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:46.844703    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:47.072309 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:47.072275    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:47.695984 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:47.695951    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:47.929555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:47.929517    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:48.155302 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:48.155261    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:49.253325 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.253281    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"]
Oct 31 19:49:49.258187 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.258160    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.306288 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.306241    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"]
Oct 31 19:49:49.312056 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.312030    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.322396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.322369    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.322548 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.322402    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.322645 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.322536    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8bz\" (UniqueName: \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-kube-api-access-mm8bz\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.322645 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.322586    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.322749 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.322701    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.322749 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.322730    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.322844 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.322830    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.322897 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.322884    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.322986 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.322975    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.354471 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.354429    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"]
Oct 31 19:49:49.405085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.405049    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"]
Oct 31 19:49:49.424196 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424148    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.424196 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424186    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4299d2a8-888c-44c5-89d2-98275371afc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.424476 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424285    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.424476 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424364    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdhs\" (UniqueName: \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-kube-api-access-bzdhs\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.424595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424514    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.424595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424554    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.424729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424611    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.424729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424675    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8bz\" (UniqueName: \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-kube-api-access-mm8bz\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.424729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424710    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.424900 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424741    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4299d2a8-888c-44c5-89d2-98275371afc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.424900 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424809    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.424900 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424840    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.425055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424901    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.425055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.424940    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.425055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.425016    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.425221 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.425107    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.425221 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.425160    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.425221 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.425189    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.426130 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.425898    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.426681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.426287    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.426681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.426419    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.426681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.426620    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.426917 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.426724    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.428881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.428853    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.429194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.429171    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.483693 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.483643    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.484001 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.483974    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8bz\" (UniqueName: \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-kube-api-access-mm8bz\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5dxr\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.526270 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526176    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526270 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526231    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526296    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526334    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4299d2a8-888c-44c5-89d2-98275371afc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526383    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526440    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdhs\" (UniqueName: \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-kube-api-access-bzdhs\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526506    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526559    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4299d2a8-888c-44c5-89d2-98275371afc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526682    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526778    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526885 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526842    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.526989 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.526940    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.527158 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.527134    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.527256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.527233    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4299d2a8-888c-44c5-89d2-98275371afc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.529389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.529364    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.529624 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.529606    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4299d2a8-888c-44c5-89d2-98275371afc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.574122 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.574079    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:49.584963 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.584935    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.585098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.585054    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdhs\" (UniqueName: \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-kube-api-access-bzdhs\") pod \"maas-default-gateway-openshift-default-58d898bd55-jrqmx\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:49.627474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:49.627426    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:50.003991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.003956    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"]
Oct 31 19:49:50.034234 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:49:50.034197    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded67b158_b0cf_4b1b_bb99_ba78733f2390.slice/crio-c38e274699973a3dfc6ebcc55554811f6f4789a275d75c644fb7d405e320d94f WatchSource:0}: Error finding container c38e274699973a3dfc6ebcc55554811f6f4789a275d75c644fb7d405e320d94f: Status 404 returned error can't find the container with id c38e274699973a3dfc6ebcc55554811f6f4789a275d75c644fb7d405e320d94f
Oct 31 19:49:50.038079 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.038030    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:50.038204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.038134    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:50.038204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.038182    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:50.054071 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:49:50.054033    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4299d2a8_888c_44c5_89d2_98275371afc7.slice/crio-52468f683cba0db4ff547d22b099ada2737fbd343af825b969c88c1d1d6133bb WatchSource:0}: Error finding container 52468f683cba0db4ff547d22b099ada2737fbd343af825b969c88c1d1d6133bb: Status 404 returned error can't find the container with id 52468f683cba0db4ff547d22b099ada2737fbd343af825b969c88c1d1d6133bb
Oct 31 19:49:50.056344 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.056317    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"]
Oct 31 19:49:50.057254 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.057213    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:50.057384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.057298    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:50.057384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.057341    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:50.304729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.303835    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" event={"ID":"ed67b158-b0cf-4b1b-bb99-ba78733f2390","Type":"ContainerStarted","Data":"c81a12347465d63d028daf1fca937429ebac49b0af147cd4e07b91051596aecc"}
Oct 31 19:49:50.304729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.303926    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" event={"ID":"ed67b158-b0cf-4b1b-bb99-ba78733f2390","Type":"ContainerStarted","Data":"c38e274699973a3dfc6ebcc55554811f6f4789a275d75c644fb7d405e320d94f"}
Oct 31 19:49:50.306625 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.306579    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" event={"ID":"4299d2a8-888c-44c5-89d2-98275371afc7","Type":"ContainerStarted","Data":"2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981"}
Oct 31 19:49:50.306852 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.306706    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" event={"ID":"4299d2a8-888c-44c5-89d2-98275371afc7","Type":"ContainerStarted","Data":"52468f683cba0db4ff547d22b099ada2737fbd343af825b969c88c1d1d6133bb"}
Oct 31 19:49:50.432812 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.432743    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" podStartSLOduration=1.432721202 podStartE2EDuration="1.432721202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:50.42198227 +0000 UTC m=+2008.951516362" watchObservedRunningTime="2026-10-31 19:49:50.432721202 +0000 UTC m=+2008.962255257"
Oct 31 19:49:50.542088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.542025    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" podStartSLOduration=1.5420082069999999 podStartE2EDuration="1.542008207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:50.53988702 +0000 UTC m=+2009.069421092" watchObservedRunningTime="2026-10-31 19:49:50.542008207 +0000 UTC m=+2009.071542328"
Oct 31 19:49:50.574705 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.574587    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:50.627921 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:50.627875    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:51.576714 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:51.576676    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5dxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:49:51.579204 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:51.576751    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 19:49:51.634237 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:51.634195    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jrqmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:49:51.637847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:51.637804    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 31 19:49:52.574700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:52.574630    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5dxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:49:52.574886 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:52.574723    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 19:49:52.647139 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:52.647100    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jrqmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:49:52.647801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:52.647764    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 31 19:49:53.173475 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:53.173116    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:53.575104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:53.575014    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5dxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:49:53.575104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:53.575071    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 19:49:53.635700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:53.635640    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jrqmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:49:53.635983 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:53.635953    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 31 19:49:54.575010 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:54.574966    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5dxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:49:54.575449 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:54.575039    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 19:49:54.630956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:54.630916    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jrqmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:49:54.631246 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:54.631216    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 31 19:49:54.931684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:54.930141    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:55.147867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:55.147827    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:55.574950 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:55.574915    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5dxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:49:55.575135 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:55.574983    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 19:49:55.638677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:55.635426    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jrqmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:49:55.638677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:55.635499    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 31 19:49:56.575362 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:56.575317    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5dxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:49:56.575854 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:56.575388    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 19:49:56.631319 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:56.631284    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jrqmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:49:56.631643 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:56.631610    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 31 19:49:57.575505 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:57.575461    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5dxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:49:57.575952 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:57.575534    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 19:49:57.628382 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:57.628342    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jrqmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:49:57.628636 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:57.628428    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 31 19:49:58.110691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:58.107807    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:58.575354 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:58.575259    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5dxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:49:58.575354 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:58.575322    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 19:49:58.628516 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:58.628468    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jrqmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:49:58.628711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:58.628535    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 31 19:49:59.627270 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:59.627230    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:59.628183 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:59.628157    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:59.629256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:59.629225    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 19:49:59.632950 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:59.632924    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:59.633233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:59.633216    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:49:59.633893 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:49:59.633876    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 19:50:03.078344 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:03.078237    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:05.395102 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:50:05.395064    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:50:07.912498 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:07.912457    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:12.799309 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:12.799269    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:17.356355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:17.356309    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:50:17.828928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:17.828772    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:22.534294 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:22.534254    5006 scope.go:122] "RemoveContainer" containerID="efd64f44a93424704d2070f9ceb9e7c1413c0ffa11256b5a4fddd21e23e9ca11"
Oct 31 19:50:23.007456 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:23.007416    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:23.221416 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:23.221379    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:28.281315 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:28.281280    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:31.631423 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:31.631383    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:31.991914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:31.991823    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"]
Oct 31 19:50:31.998029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:31.998004    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.051673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.051621    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"]
Oct 31 19:50:32.095433 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.095383    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:50:32.095433 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.095412    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:50:32.095673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.095551    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:50:32.101098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.101067    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:50:32.184927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.184888    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglrp\" (UniqueName: \"kubernetes.io/projected/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-kube-api-access-hglrp\") pod \"maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.185161 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.184947    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.185161 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.185038    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.286585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.286497    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglrp\" (UniqueName: \"kubernetes.io/projected/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-kube-api-access-hglrp\") pod \"maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.286585 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.286574    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.286837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.286621    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.287507 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.287474    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.289176 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.289141    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.344348 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.344310    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglrp\" (UniqueName: \"kubernetes.io/projected/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-kube-api-access-hglrp\") pod \"maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.612864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.612828    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:32.754491 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.754451    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:32.975377 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:32.975291    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:33.020118 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:50:33.020082    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1adc564_79bd_44d2_9700_0f0ddd9a3e90.slice/crio-cca02d6fcecf45cea20b2466a17a554b9d0abdc3bddc70f5ea8fd00b11be4668 WatchSource:0}: Error finding container cca02d6fcecf45cea20b2466a17a554b9d0abdc3bddc70f5ea8fd00b11be4668: Status 404 returned error can't find the container with id cca02d6fcecf45cea20b2466a17a554b9d0abdc3bddc70f5ea8fd00b11be4668
Oct 31 19:50:33.027335 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:33.027307    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"]
Oct 31 19:50:33.635498 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:33.635464    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:33.809948 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:33.809859    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp" event={"ID":"d1adc564-79bd-44d2-9700-0f0ddd9a3e90","Type":"ContainerStarted","Data":"b3b1cc74b6ddf59d6fa9dcaec5ef246a68dd084ddd8184be0ac30cea58b694ff"}
Oct 31 19:50:33.809948 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:33.809930    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp" event={"ID":"d1adc564-79bd-44d2-9700-0f0ddd9a3e90","Type":"ContainerStarted","Data":"cca02d6fcecf45cea20b2466a17a554b9d0abdc3bddc70f5ea8fd00b11be4668"}
Oct 31 19:50:33.927322 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:33.927263    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp" podStartSLOduration=2.927247797 podStartE2EDuration="2.927247797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:33.922755022 +0000 UTC m=+2052.452289075" watchObservedRunningTime="2026-10-31 19:50:33.927247797 +0000 UTC m=+2052.456781850"
Oct 31 19:50:34.819998 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:34.819961    5006 generic.go:356] "Generic (PLEG): container finished" podID="d1adc564-79bd-44d2-9700-0f0ddd9a3e90" containerID="b3b1cc74b6ddf59d6fa9dcaec5ef246a68dd084ddd8184be0ac30cea58b694ff" exitCode=0
Oct 31 19:50:34.820434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:34.820048    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp" event={"ID":"d1adc564-79bd-44d2-9700-0f0ddd9a3e90","Type":"ContainerDied","Data":"b3b1cc74b6ddf59d6fa9dcaec5ef246a68dd084ddd8184be0ac30cea58b694ff"}
Oct 31 19:50:35.404567 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:50:35.404518    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:50:36.008143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.008116    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:36.045097 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.045054    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-kube-api-access-hglrp\" (UniqueName: \"kubernetes.io/projected/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-kube-api-access-hglrp\") pod \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") "
Oct 31 19:50:36.045310 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.045293    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-openshift-ca\") pod \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") "
Oct 31 19:50:36.045379 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.045367    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-xks-ca\") pod \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\" (UID: \"d1adc564-79bd-44d2-9700-0f0ddd9a3e90\") "
Oct 31 19:50:36.045684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.045639    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-openshift-ca" pod "d1adc564-79bd-44d2-9700-0f0ddd9a3e90" (UID: "d1adc564-79bd-44d2-9700-0f0ddd9a3e90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:36.045962 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.045942    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:36.047393 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.047362    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-xks-ca" pod "d1adc564-79bd-44d2-9700-0f0ddd9a3e90" (UID: "d1adc564-79bd-44d2-9700-0f0ddd9a3e90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:36.047489 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.047466    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-kube-api-access-hglrp" pod "d1adc564-79bd-44d2-9700-0f0ddd9a3e90" (UID: "d1adc564-79bd-44d2-9700-0f0ddd9a3e90"). InnerVolumeSpecName "kube-api-access-hglrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:36.147783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.147738    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:36.147783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.147784    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hglrp\" (UniqueName: \"kubernetes.io/projected/d1adc564-79bd-44d2-9700-0f0ddd9a3e90-kube-api-access-hglrp\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:36.842091 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.842056    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"
Oct 31 19:50:36.842091 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.842085    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp" event={"ID":"d1adc564-79bd-44d2-9700-0f0ddd9a3e90","Type":"ContainerDied","Data":"cca02d6fcecf45cea20b2466a17a554b9d0abdc3bddc70f5ea8fd00b11be4668"}
Oct 31 19:50:36.842316 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:36.842139    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cca02d6fcecf45cea20b2466a17a554b9d0abdc3bddc70f5ea8fd00b11be4668"
Oct 31 19:50:38.897483 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:38.897442    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"]
Oct 31 19:50:38.898281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:38.898257    5006 cpu_manager.go:414] "containerMap: removing container" podUID="d1adc564-79bd-44d2-9700-0f0ddd9a3e90" containerName="revoke-keys"
Oct 31 19:50:38.898281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:38.898280    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1adc564-79bd-44d2-9700-0f0ddd9a3e90" containerName="revoke-keys"
Oct 31 19:50:38.898473 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:38.898449    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1adc564-79bd-44d2-9700-0f0ddd9a3e90" containerName="revoke-keys"
Oct 31 19:50:38.902842 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:38.902812    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:38.954512 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:38.954471    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"]
Oct 31 19:50:38.993472 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:38.993421    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa988fb4-jc8pj\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:38.993472 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:38.993485    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa988fb4-jc8pj\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:38.993886 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:38.993746    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsjg7\" (UniqueName: \"kubernetes.io/projected/3a37408c-91cc-4cf4-b17c-cea5908dfd47-kube-api-access-fsjg7\") pod \"maas-api-revoke-sub-aa988fb4-jc8pj\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:38.998783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:38.998749    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:50:39.006405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.006371    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:50:39.006628 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.006414    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:50:39.006628 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.006526    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:50:39.095603 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.095536    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa988fb4-jc8pj\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:39.095847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.095622    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa988fb4-jc8pj\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:39.095847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.095806    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsjg7\" (UniqueName: \"kubernetes.io/projected/3a37408c-91cc-4cf4-b17c-cea5908dfd47-kube-api-access-fsjg7\") pod \"maas-api-revoke-sub-aa988fb4-jc8pj\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:39.096344 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.096319    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa988fb4-jc8pj\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:39.098142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.098115    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa988fb4-jc8pj\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:39.168729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.163476    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsjg7\" (UniqueName: \"kubernetes.io/projected/3a37408c-91cc-4cf4-b17c-cea5908dfd47-kube-api-access-fsjg7\") pod \"maas-api-revoke-sub-aa988fb4-jc8pj\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:39.168729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.165462    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:39.220119 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.220082    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:39.606547 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:50:39.606515    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a37408c_91cc_4cf4_b17c_cea5908dfd47.slice/crio-859385cc9d21c4f24fd6e52ef69a98ffba02f06df9ff3f139eeed66d5bf09e48 WatchSource:0}: Error finding container 859385cc9d21c4f24fd6e52ef69a98ffba02f06df9ff3f139eeed66d5bf09e48: Status 404 returned error can't find the container with id 859385cc9d21c4f24fd6e52ef69a98ffba02f06df9ff3f139eeed66d5bf09e48
Oct 31 19:50:39.609417 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.609391    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"]
Oct 31 19:50:39.867592 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:39.867500    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj" event={"ID":"3a37408c-91cc-4cf4-b17c-cea5908dfd47","Type":"ContainerStarted","Data":"859385cc9d21c4f24fd6e52ef69a98ffba02f06df9ff3f139eeed66d5bf09e48"}
Oct 31 19:50:40.878730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:40.878695    5006 generic.go:356] "Generic (PLEG): container finished" podID="3a37408c-91cc-4cf4-b17c-cea5908dfd47" containerID="0f468ecec751b8561e4a750592261d2e1aeb94e1802d6b3b550002abd27bc00c" exitCode=0
Oct 31 19:50:40.879148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:40.878781    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj" event={"ID":"3a37408c-91cc-4cf4-b17c-cea5908dfd47","Type":"ContainerDied","Data":"0f468ecec751b8561e4a750592261d2e1aeb94e1802d6b3b550002abd27bc00c"}
Oct 31 19:50:41.060976 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:41.060936    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"]
Oct 31 19:50:41.067084 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:41.067051    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9993bb24-t2d9c"]
Oct 31 19:50:41.603078 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:41.603040    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c44c19-46af-4b3a-b2c0-57cda3fb8b60" path="/var/lib/kubelet/pods/a8c44c19-46af-4b3a-b2c0-57cda3fb8b60/volumes"
Oct 31 19:50:42.075346 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.075313    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:42.143274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.143185    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-openshift-ca\") pod \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") "
Oct 31 19:50:42.143498 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.143483    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-xks-ca\") pod \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") "
Oct 31 19:50:42.143561 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.143507    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a37408c-91cc-4cf4-b17c-cea5908dfd47-kube-api-access-fsjg7\" (UniqueName: \"kubernetes.io/projected/3a37408c-91cc-4cf4-b17c-cea5908dfd47-kube-api-access-fsjg7\") pod \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\" (UID: \"3a37408c-91cc-4cf4-b17c-cea5908dfd47\") "
Oct 31 19:50:42.143625 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.143587    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-openshift-ca" pod "3a37408c-91cc-4cf4-b17c-cea5908dfd47" (UID: "3a37408c-91cc-4cf4-b17c-cea5908dfd47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:42.144058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.144034    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:42.145793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.145762    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-xks-ca" pod "3a37408c-91cc-4cf4-b17c-cea5908dfd47" (UID: "3a37408c-91cc-4cf4-b17c-cea5908dfd47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:42.145920 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.145763    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a37408c-91cc-4cf4-b17c-cea5908dfd47-kube-api-access-fsjg7" pod "3a37408c-91cc-4cf4-b17c-cea5908dfd47" (UID: "3a37408c-91cc-4cf4-b17c-cea5908dfd47"). InnerVolumeSpecName "kube-api-access-fsjg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:42.246112 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.246059    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a37408c-91cc-4cf4-b17c-cea5908dfd47-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:42.246112 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.246116    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsjg7\" (UniqueName: \"kubernetes.io/projected/3a37408c-91cc-4cf4-b17c-cea5908dfd47-kube-api-access-fsjg7\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:42.899003 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.898966    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"
Oct 31 19:50:42.899003 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.898988    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj" event={"ID":"3a37408c-91cc-4cf4-b17c-cea5908dfd47","Type":"ContainerDied","Data":"859385cc9d21c4f24fd6e52ef69a98ffba02f06df9ff3f139eeed66d5bf09e48"}
Oct 31 19:50:42.899223 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.899044    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="859385cc9d21c4f24fd6e52ef69a98ffba02f06df9ff3f139eeed66d5bf09e48"
Oct 31 19:50:42.929930 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:42.929894    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"]
Oct 31 19:50:43.041954 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.041918    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"]
Oct 31 19:50:43.378402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.378365    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:43.604577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.604523    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:43.927926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.927888    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"]
Oct 31 19:50:43.928826 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.928805    5006 cpu_manager.go:414] "containerMap: removing container" podUID="3a37408c-91cc-4cf4-b17c-cea5908dfd47" containerName="revoke-keys"
Oct 31 19:50:43.928914 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.928826    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a37408c-91cc-4cf4-b17c-cea5908dfd47" containerName="revoke-keys"
Oct 31 19:50:43.928959 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.928945    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a37408c-91cc-4cf4-b17c-cea5908dfd47" containerName="revoke-keys"
Oct 31 19:50:43.932927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.932908    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:43.990572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.985862    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:50:43.990572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.985923    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:50:43.990572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.985985    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:50:43.990572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.986039    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:50:43.990572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:43.988073    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"]
Oct 31 19:50:44.090099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.090061    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae778663-rvgqg\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:44.090280 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.090186    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxlp\" (UniqueName: \"kubernetes.io/projected/c2927278-9741-4e00-adf5-a1457ebd9c18-kube-api-access-phxlp\") pod \"maas-api-revoke-keys-e2e-ait-ae778663-rvgqg\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:44.090326 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.090307    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae778663-rvgqg\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:44.192089 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.191991    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae778663-rvgqg\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:44.192265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.192105    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxlp\" (UniqueName: \"kubernetes.io/projected/c2927278-9741-4e00-adf5-a1457ebd9c18-kube-api-access-phxlp\") pod \"maas-api-revoke-keys-e2e-ait-ae778663-rvgqg\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:44.192265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.192195    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae778663-rvgqg\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:44.192887 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.192859    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae778663-rvgqg\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:44.194601 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.194580    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae778663-rvgqg\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:44.250835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.250803    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxlp\" (UniqueName: \"kubernetes.io/projected/c2927278-9741-4e00-adf5-a1457ebd9c18-kube-api-access-phxlp\") pod \"maas-api-revoke-keys-e2e-ait-ae778663-rvgqg\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:44.500736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.499822    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:44.559961 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:44.559920    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:45.043712 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:50:45.041694    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2927278_9741_4e00_adf5_a1457ebd9c18.slice/crio-8aa2fe020bca3e9ac397de9c51f161b5d507f8a1e8e7d0987a71e4ab2497885b WatchSource:0}: Error finding container 8aa2fe020bca3e9ac397de9c51f161b5d507f8a1e8e7d0987a71e4ab2497885b: Status 404 returned error can't find the container with id 8aa2fe020bca3e9ac397de9c51f161b5d507f8a1e8e7d0987a71e4ab2497885b
Oct 31 19:50:45.051172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:45.048362    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"]
Oct 31 19:50:45.928516 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:45.928473    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg" event={"ID":"c2927278-9741-4e00-adf5-a1457ebd9c18","Type":"ContainerStarted","Data":"e8a56c087038cc9d21abf9649c0c59b5d1f7f3c65d6fb5af33adbb26c62048ee"}
Oct 31 19:50:45.928963 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:45.928538    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg" event={"ID":"c2927278-9741-4e00-adf5-a1457ebd9c18","Type":"ContainerStarted","Data":"8aa2fe020bca3e9ac397de9c51f161b5d507f8a1e8e7d0987a71e4ab2497885b"}
Oct 31 19:50:46.053520 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:46.053451    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg" podStartSLOduration=3.053430356 podStartE2EDuration="3.053430356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:46.046040149 +0000 UTC m=+2064.575574222" watchObservedRunningTime="2026-10-31 19:50:46.053430356 +0000 UTC m=+2064.582964408"
Oct 31 19:50:46.939378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:46.939335    5006 generic.go:356] "Generic (PLEG): container finished" podID="c2927278-9741-4e00-adf5-a1457ebd9c18" containerID="e8a56c087038cc9d21abf9649c0c59b5d1f7f3c65d6fb5af33adbb26c62048ee" exitCode=0
Oct 31 19:50:46.939806 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:46.939422    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg" event={"ID":"c2927278-9741-4e00-adf5-a1457ebd9c18","Type":"ContainerDied","Data":"e8a56c087038cc9d21abf9649c0c59b5d1f7f3c65d6fb5af33adbb26c62048ee"}
Oct 31 19:50:47.930246 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:47.930184    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h" podUID="4fd2e91d-668e-4bf6-9ecd-f95b961a20e1" containerName="payload-processing" containerID="cri-o://c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7" gracePeriod=25
Oct 31 19:50:48.043150 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.043106    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln" podUID="3249705a-b2e0-4b8f-8757-661f9b936732" containerName="payload-pre-processing" containerID="cri-o://96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0" gracePeriod=25
Oct 31 19:50:48.204745 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.204611    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:48.248968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.248936    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:50:48.349303 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.349273    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:50:48.367555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.367518    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-plugins-config-volume\") pod \"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1\" (UID: \"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1\") "
Oct 31 19:50:48.367801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.367615    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-openshift-ca\") pod \"c2927278-9741-4e00-adf5-a1457ebd9c18\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") "
Oct 31 19:50:48.367801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.367669    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2927278-9741-4e00-adf5-a1457ebd9c18-kube-api-access-phxlp\" (UniqueName: \"kubernetes.io/projected/c2927278-9741-4e00-adf5-a1457ebd9c18-kube-api-access-phxlp\") pod \"c2927278-9741-4e00-adf5-a1457ebd9c18\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") "
Oct 31 19:50:48.367801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.367781    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-kube-api-access-kzqkj\" (UniqueName: \"kubernetes.io/projected/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-kube-api-access-kzqkj\") pod \"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1\" (UID: \"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1\") "
Oct 31 19:50:48.368043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.367821    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-xks-ca\") pod \"c2927278-9741-4e00-adf5-a1457ebd9c18\" (UID: \"c2927278-9741-4e00-adf5-a1457ebd9c18\") "
Oct 31 19:50:48.369009 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.368973    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-openshift-ca" pod "c2927278-9741-4e00-adf5-a1457ebd9c18" (UID: "c2927278-9741-4e00-adf5-a1457ebd9c18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:48.369144 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.369007    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-plugins-config-volume" pod "4fd2e91d-668e-4bf6-9ecd-f95b961a20e1" (UID: "4fd2e91d-668e-4bf6-9ecd-f95b961a20e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:48.371021 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.370988    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-xks-ca" pod "c2927278-9741-4e00-adf5-a1457ebd9c18" (UID: "c2927278-9741-4e00-adf5-a1457ebd9c18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:48.371768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.371731    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-kube-api-access-kzqkj" pod "4fd2e91d-668e-4bf6-9ecd-f95b961a20e1" (UID: "4fd2e91d-668e-4bf6-9ecd-f95b961a20e1"). InnerVolumeSpecName "kube-api-access-kzqkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:48.371884 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.371865    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2927278-9741-4e00-adf5-a1457ebd9c18-kube-api-access-phxlp" pod "c2927278-9741-4e00-adf5-a1457ebd9c18" (UID: "c2927278-9741-4e00-adf5-a1457ebd9c18"). InnerVolumeSpecName "kube-api-access-phxlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:48.469629 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.469520    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3249705a-b2e0-4b8f-8757-661f9b936732-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3249705a-b2e0-4b8f-8757-661f9b936732-plugins-config-volume\") pod \"3249705a-b2e0-4b8f-8757-661f9b936732\" (UID: \"3249705a-b2e0-4b8f-8757-661f9b936732\") "
Oct 31 19:50:48.469813 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.469698    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3249705a-b2e0-4b8f-8757-661f9b936732-kube-api-access-88kvt\" (UniqueName: \"kubernetes.io/projected/3249705a-b2e0-4b8f-8757-661f9b936732-kube-api-access-88kvt\") pod \"3249705a-b2e0-4b8f-8757-661f9b936732\" (UID: \"3249705a-b2e0-4b8f-8757-661f9b936732\") "
Oct 31 19:50:48.470013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.469989    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3249705a-b2e0-4b8f-8757-661f9b936732-plugins-config-volume" pod "3249705a-b2e0-4b8f-8757-661f9b936732" (UID: "3249705a-b2e0-4b8f-8757-661f9b936732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:48.470312 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.470285    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzqkj\" (UniqueName: \"kubernetes.io/projected/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-kube-api-access-kzqkj\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:48.470312 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.470307    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:48.470470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.470323    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3249705a-b2e0-4b8f-8757-661f9b936732-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:48.470470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.470337    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:48.470470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.470349    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2927278-9741-4e00-adf5-a1457ebd9c18-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:48.470470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.470364    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phxlp\" (UniqueName: \"kubernetes.io/projected/c2927278-9741-4e00-adf5-a1457ebd9c18-kube-api-access-phxlp\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:48.471896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.471873    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3249705a-b2e0-4b8f-8757-661f9b936732-kube-api-access-88kvt" pod "3249705a-b2e0-4b8f-8757-661f9b936732" (UID: "3249705a-b2e0-4b8f-8757-661f9b936732"). InnerVolumeSpecName "kube-api-access-88kvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:48.571842 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.571798    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88kvt\" (UniqueName: \"kubernetes.io/projected/3249705a-b2e0-4b8f-8757-661f9b936732-kube-api-access-88kvt\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:48.959884 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.959835    5006 generic.go:356] "Generic (PLEG): container finished" podID="4fd2e91d-668e-4bf6-9ecd-f95b961a20e1" containerID="c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7" exitCode=0
Oct 31 19:50:48.960259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.960216    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h" event={"ID":"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1","Type":"ContainerDied","Data":"c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7"}
Oct 31 19:50:48.963440 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.960277    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h" event={"ID":"4fd2e91d-668e-4bf6-9ecd-f95b961a20e1","Type":"ContainerDied","Data":"61a39d3f5c88bc28c157554b1a2b5d68c88fc468ea2bd02da8979278247f92f9"}
Oct 31 19:50:48.963440 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.960309    5006 scope.go:122] "RemoveContainer" containerID="c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7"
Oct 31 19:50:48.963440 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:48.960484    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"
Oct 31 19:50:49.004687 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.004304    5006 generic.go:356] "Generic (PLEG): container finished" podID="3249705a-b2e0-4b8f-8757-661f9b936732" containerID="96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0" exitCode=0
Oct 31 19:50:49.004687 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.004545    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln" event={"ID":"3249705a-b2e0-4b8f-8757-661f9b936732","Type":"ContainerDied","Data":"96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0"}
Oct 31 19:50:49.004687 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.004614    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln" event={"ID":"3249705a-b2e0-4b8f-8757-661f9b936732","Type":"ContainerDied","Data":"4890c0a15261e00c515dc4a1bf55cb5b73f986c5fa9c6c2d7af66e2144569f61"}
Oct 31 19:50:49.005012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.004760    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"
Oct 31 19:50:49.078429 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.078392    5006 scope.go:122] "RemoveContainer" containerID="c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7"
Oct 31 19:50:49.081044 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:50:49.081008    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7\": container with ID starting with c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7 not found: ID does not exist" containerID="c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7"
Oct 31 19:50:49.081243 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.081216    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7"} err="failed to get container status \"c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7\": rpc error: code = NotFound desc = could not find container \"c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7\": container with ID starting with c41feafd4d623f0c48e8c605e086feae1dead26eb46bf7cf9c81ab1a450142a7 not found: ID does not exist"
Oct 31 19:50:49.081354 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.081343    5006 scope.go:122] "RemoveContainer" containerID="96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0"
Oct 31 19:50:49.081759 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.081734    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg" event={"ID":"c2927278-9741-4e00-adf5-a1457ebd9c18","Type":"ContainerDied","Data":"8aa2fe020bca3e9ac397de9c51f161b5d507f8a1e8e7d0987a71e4ab2497885b"}
Oct 31 19:50:49.081918 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.081903    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aa2fe020bca3e9ac397de9c51f161b5d507f8a1e8e7d0987a71e4ab2497885b"
Oct 31 19:50:49.082070 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.082059    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"
Oct 31 19:50:49.119692 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.119596    5006 scope.go:122] "RemoveContainer" containerID="96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0"
Oct 31 19:50:49.120257 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:50:49.120170    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0\": container with ID starting with 96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0 not found: ID does not exist" containerID="96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0"
Oct 31 19:50:49.120257 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.120206    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0"} err="failed to get container status \"96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0\": rpc error: code = NotFound desc = could not find container \"96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0\": container with ID starting with 96cf4691db860e17ee69ad2973daf3cd4edf3e5687140b292bb1c8bb1e8ce7d0 not found: ID does not exist"
Oct 31 19:50:49.233018 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.232928    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"]
Oct 31 19:50:49.238568 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.238508    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9be7bd13-67d78455c-sv68h"]
Oct 31 19:50:49.346020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.345982    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"]
Oct 31 19:50:49.351312 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.351280    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9be7bd13-bd4d98c8c-69qln"]
Oct 31 19:50:49.601781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.601743    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3249705a-b2e0-4b8f-8757-661f9b936732" path="/var/lib/kubelet/pods/3249705a-b2e0-4b8f-8757-661f9b936732/volumes"
Oct 31 19:50:49.602435 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.602415    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd2e91d-668e-4bf6-9ecd-f95b961a20e1" path="/var/lib/kubelet/pods/4fd2e91d-668e-4bf6-9ecd-f95b961a20e1/volumes"
Oct 31 19:50:49.744222 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:49.744188    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:50.023188 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.023097    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:50.872080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.872040    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"]
Oct 31 19:50:50.872755 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.872739    5006 cpu_manager.go:414] "containerMap: removing container" podUID="3249705a-b2e0-4b8f-8757-661f9b936732" containerName="payload-pre-processing"
Oct 31 19:50:50.872755 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.872755    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3249705a-b2e0-4b8f-8757-661f9b936732" containerName="payload-pre-processing"
Oct 31 19:50:50.872863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.872776    5006 cpu_manager.go:414] "containerMap: removing container" podUID="4fd2e91d-668e-4bf6-9ecd-f95b961a20e1" containerName="payload-processing"
Oct 31 19:50:50.872863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.872782    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd2e91d-668e-4bf6-9ecd-f95b961a20e1" containerName="payload-processing"
Oct 31 19:50:50.872863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.872794    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c2927278-9741-4e00-adf5-a1457ebd9c18" containerName="revoke-keys"
Oct 31 19:50:50.872863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.872800    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2927278-9741-4e00-adf5-a1457ebd9c18" containerName="revoke-keys"
Oct 31 19:50:50.873000 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.872929    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3249705a-b2e0-4b8f-8757-661f9b936732" containerName="payload-pre-processing"
Oct 31 19:50:50.873000 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.872940    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd2e91d-668e-4bf6-9ecd-f95b961a20e1" containerName="payload-processing"
Oct 31 19:50:50.873000 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.872947    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2927278-9741-4e00-adf5-a1457ebd9c18" containerName="revoke-keys"
Oct 31 19:50:50.878352 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.878329    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:50.932256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.932219    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"]
Oct 31 19:50:50.976881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.976841    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:50:50.976881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.976849    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:50:50.977110 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.976913    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:50:50.982068 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:50.982036    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:50:51.016484 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.016449    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h57f\" (UniqueName: \"kubernetes.io/projected/9a110888-e4b9-4b17-a71a-72187634d3b4-kube-api-access-5h57f\") pod \"maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:51.016724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.016619    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:51.016879 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.016857    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:51.118800 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.118745    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:51.118951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.118862    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h57f\" (UniqueName: \"kubernetes.io/projected/9a110888-e4b9-4b17-a71a-72187634d3b4-kube-api-access-5h57f\") pod \"maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:51.118951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.118945    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:51.119642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.119617    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:51.121536 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.121502    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:51.176968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.176888    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h57f\" (UniqueName: \"kubernetes.io/projected/9a110888-e4b9-4b17-a71a-72187634d3b4-kube-api-access-5h57f\") pod \"maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:51.194517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.194486    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:51.574123 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:50:51.573884    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a110888_e4b9_4b17_a71a_72187634d3b4.slice/crio-0b42f5c399f29b1101a1c16372a9db2c918898d7d8034f345a1f0d9d87d9014c WatchSource:0}: Error finding container 0b42f5c399f29b1101a1c16372a9db2c918898d7d8034f345a1f0d9d87d9014c: Status 404 returned error can't find the container with id 0b42f5c399f29b1101a1c16372a9db2c918898d7d8034f345a1f0d9d87d9014c
Oct 31 19:50:51.582034 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:51.582001    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"]
Oct 31 19:50:52.113165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:52.113117    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69" event={"ID":"9a110888-e4b9-4b17-a71a-72187634d3b4","Type":"ContainerStarted","Data":"ede1f603a77bd5e561cbe8eeb1adc20878fd514922905884b2e44e1d07e01f89"}
Oct 31 19:50:52.113566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:52.113179    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69" event={"ID":"9a110888-e4b9-4b17-a71a-72187634d3b4","Type":"ContainerStarted","Data":"0b42f5c399f29b1101a1c16372a9db2c918898d7d8034f345a1f0d9d87d9014c"}
Oct 31 19:50:52.231979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:52.231921    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69" podStartSLOduration=2.2319029 podStartE2EDuration="2.2319029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:52.228104404 +0000 UTC m=+2070.757638484" watchObservedRunningTime="2026-10-31 19:50:52.2319029 +0000 UTC m=+2070.761436952"
Oct 31 19:50:53.123590 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:53.123557    5006 generic.go:356] "Generic (PLEG): container finished" podID="9a110888-e4b9-4b17-a71a-72187634d3b4" containerID="ede1f603a77bd5e561cbe8eeb1adc20878fd514922905884b2e44e1d07e01f89" exitCode=0
Oct 31 19:50:53.124058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:53.123642    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69" event={"ID":"9a110888-e4b9-4b17-a71a-72187634d3b4","Type":"ContainerDied","Data":"ede1f603a77bd5e561cbe8eeb1adc20878fd514922905884b2e44e1d07e01f89"}
Oct 31 19:50:54.333295 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:54.333265    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:54.368823 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:54.368788    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a110888-e4b9-4b17-a71a-72187634d3b4-kube-api-access-5h57f\" (UniqueName: \"kubernetes.io/projected/9a110888-e4b9-4b17-a71a-72187634d3b4-kube-api-access-5h57f\") pod \"9a110888-e4b9-4b17-a71a-72187634d3b4\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") "
Oct 31 19:50:54.368983 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:54.368862    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-xks-ca\") pod \"9a110888-e4b9-4b17-a71a-72187634d3b4\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") "
Oct 31 19:50:54.368983 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:54.368963    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-openshift-ca\") pod \"9a110888-e4b9-4b17-a71a-72187634d3b4\" (UID: \"9a110888-e4b9-4b17-a71a-72187634d3b4\") "
Oct 31 19:50:54.369300 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:54.369269    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-openshift-ca" pod "9a110888-e4b9-4b17-a71a-72187634d3b4" (UID: "9a110888-e4b9-4b17-a71a-72187634d3b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:54.369791 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:54.369773    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.371184 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:54.371154    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-xks-ca" pod "9a110888-e4b9-4b17-a71a-72187634d3b4" (UID: "9a110888-e4b9-4b17-a71a-72187634d3b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:54.371266 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:54.371210    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a110888-e4b9-4b17-a71a-72187634d3b4-kube-api-access-5h57f" pod "9a110888-e4b9-4b17-a71a-72187634d3b4" (UID: "9a110888-e4b9-4b17-a71a-72187634d3b4"). InnerVolumeSpecName "kube-api-access-5h57f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:54.471423 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:54.471331    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h57f\" (UniqueName: \"kubernetes.io/projected/9a110888-e4b9-4b17-a71a-72187634d3b4-kube-api-access-5h57f\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.471423 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:54.471367    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a110888-e4b9-4b17-a71a-72187634d3b4-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:55.142584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:55.142546    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69" event={"ID":"9a110888-e4b9-4b17-a71a-72187634d3b4","Type":"ContainerDied","Data":"0b42f5c399f29b1101a1c16372a9db2c918898d7d8034f345a1f0d9d87d9014c"}
Oct 31 19:50:55.142584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:55.142567    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"
Oct 31 19:50:55.142832 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:55.142626    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b42f5c399f29b1101a1c16372a9db2c918898d7d8034f345a1f0d9d87d9014c"
Oct 31 19:50:55.210758 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:55.210690    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:55.477434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:55.477308    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:55.686972 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:55.686930    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"]
Oct 31 19:50:56.029750 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:56.029633    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:56.056975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:56.056914    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"]
Oct 31 19:50:56.066261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:56.066220    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651f5bab-ngp4x"]
Oct 31 19:50:56.278039 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:56.278004    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:57.603131 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:50:57.603092    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0784a054-7784-4de5-8366-20c28db11ca2" path="/var/lib/kubelet/pods/0784a054-7784-4de5-8366-20c28db11ca2/volumes"
Oct 31 19:51:00.144280 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:00.144226    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:00.688095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:00.688055    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd" podUID="6141a7b4-2875-4dad-90b8-2576b934b347" containerName="payload-pre-processing" containerID="cri-o://a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290" gracePeriod=25
Oct 31 19:51:00.988455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:00.988423    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:51:01.041601 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.041557    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6141a7b4-2875-4dad-90b8-2576b934b347-kube-api-access-n6kkf\" (UniqueName: \"kubernetes.io/projected/6141a7b4-2875-4dad-90b8-2576b934b347-kube-api-access-n6kkf\") pod \"6141a7b4-2875-4dad-90b8-2576b934b347\" (UID: \"6141a7b4-2875-4dad-90b8-2576b934b347\") "
Oct 31 19:51:01.041872 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.041731    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6141a7b4-2875-4dad-90b8-2576b934b347-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6141a7b4-2875-4dad-90b8-2576b934b347-plugins-config-volume\") pod \"6141a7b4-2875-4dad-90b8-2576b934b347\" (UID: \"6141a7b4-2875-4dad-90b8-2576b934b347\") "
Oct 31 19:51:01.042108 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.042083    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6141a7b4-2875-4dad-90b8-2576b934b347-plugins-config-volume" pod "6141a7b4-2875-4dad-90b8-2576b934b347" (UID: "6141a7b4-2875-4dad-90b8-2576b934b347"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:01.042290 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.042272    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6141a7b4-2875-4dad-90b8-2576b934b347-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:01.043866 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.043841    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6141a7b4-2875-4dad-90b8-2576b934b347-kube-api-access-n6kkf" pod "6141a7b4-2875-4dad-90b8-2576b934b347" (UID: "6141a7b4-2875-4dad-90b8-2576b934b347"). InnerVolumeSpecName "kube-api-access-n6kkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:01.144120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.144081    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6kkf\" (UniqueName: \"kubernetes.io/projected/6141a7b4-2875-4dad-90b8-2576b934b347-kube-api-access-n6kkf\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:01.240364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.240272    5006 generic.go:356] "Generic (PLEG): container finished" podID="6141a7b4-2875-4dad-90b8-2576b934b347" containerID="a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290" exitCode=0
Oct 31 19:51:01.240364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.240318    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"
Oct 31 19:51:01.240364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.240353    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd" event={"ID":"6141a7b4-2875-4dad-90b8-2576b934b347","Type":"ContainerDied","Data":"a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290"}
Oct 31 19:51:01.240936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.240409    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd" event={"ID":"6141a7b4-2875-4dad-90b8-2576b934b347","Type":"ContainerDied","Data":"9f0797bf8fe46cea89e91ce8843ec51ff8cb1101e341656fba7f6c6315d3b567"}
Oct 31 19:51:01.240936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.240431    5006 scope.go:122] "RemoveContainer" containerID="a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290"
Oct 31 19:51:01.252576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.252557    5006 scope.go:122] "RemoveContainer" containerID="a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290"
Oct 31 19:51:01.252944 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:51:01.252923    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290\": container with ID starting with a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290 not found: ID does not exist" containerID="a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290"
Oct 31 19:51:01.253005 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.252954    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290"} err="failed to get container status \"a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290\": rpc error: code = NotFound desc = could not find container \"a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290\": container with ID starting with a5729b7fec9db454f50b1df54fea82cf1bc21b341d3d4ab8321fce85788c9290 not found: ID does not exist"
Oct 31 19:51:01.412977 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.412932    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"]
Oct 31 19:51:01.419571 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.419531    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae778663-5fd85ffcdc-lxsrd"]
Oct 31 19:51:01.600406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:01.600369    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6141a7b4-2875-4dad-90b8-2576b934b347" path="/var/lib/kubelet/pods/6141a7b4-2875-4dad-90b8-2576b934b347/volumes"
Oct 31 19:51:02.471166 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:02.471125    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"]
Oct 31 19:51:02.813774 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:02.813621    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:03.031458 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:03.031422    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:05.293050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:05.293018    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:05.399165 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:51:05.399120    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:51:07.474556 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:07.472893    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8" podUID="a429faf1-b779-4be2-a8a2-ad6a71558cc0" containerName="payload-pre-processing" containerID="cri-o://b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0" gracePeriod=25
Oct 31 19:51:07.849924 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:07.849887    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:51:08.003536 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.003479    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a429faf1-b779-4be2-a8a2-ad6a71558cc0-kube-api-access-vstkm\" (UniqueName: \"kubernetes.io/projected/a429faf1-b779-4be2-a8a2-ad6a71558cc0-kube-api-access-vstkm\") pod \"a429faf1-b779-4be2-a8a2-ad6a71558cc0\" (UID: \"a429faf1-b779-4be2-a8a2-ad6a71558cc0\") "
Oct 31 19:51:08.003770 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.003549    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a429faf1-b779-4be2-a8a2-ad6a71558cc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a429faf1-b779-4be2-a8a2-ad6a71558cc0-plugins-config-volume\") pod \"a429faf1-b779-4be2-a8a2-ad6a71558cc0\" (UID: \"a429faf1-b779-4be2-a8a2-ad6a71558cc0\") "
Oct 31 19:51:08.004455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.004404    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a429faf1-b779-4be2-a8a2-ad6a71558cc0-plugins-config-volume" pod "a429faf1-b779-4be2-a8a2-ad6a71558cc0" (UID: "a429faf1-b779-4be2-a8a2-ad6a71558cc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:08.031685 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.014145    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a429faf1-b779-4be2-a8a2-ad6a71558cc0-kube-api-access-vstkm" pod "a429faf1-b779-4be2-a8a2-ad6a71558cc0" (UID: "a429faf1-b779-4be2-a8a2-ad6a71558cc0"). InnerVolumeSpecName "kube-api-access-vstkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:08.105709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.105663    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vstkm\" (UniqueName: \"kubernetes.io/projected/a429faf1-b779-4be2-a8a2-ad6a71558cc0-kube-api-access-vstkm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:08.105709 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.105708    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a429faf1-b779-4be2-a8a2-ad6a71558cc0-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:08.383729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.382178    5006 generic.go:356] "Generic (PLEG): container finished" podID="a429faf1-b779-4be2-a8a2-ad6a71558cc0" containerID="b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0" exitCode=0
Oct 31 19:51:08.383729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.382237    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"
Oct 31 19:51:08.383729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.382365    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8" event={"ID":"a429faf1-b779-4be2-a8a2-ad6a71558cc0","Type":"ContainerDied","Data":"b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0"}
Oct 31 19:51:08.383729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.382415    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8" event={"ID":"a429faf1-b779-4be2-a8a2-ad6a71558cc0","Type":"ContainerDied","Data":"97c2d7e18a96ec03fa257b3064b01136252f9e86a73c0c6983d49ba0f77fbc49"}
Oct 31 19:51:08.383729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.382445    5006 scope.go:122] "RemoveContainer" containerID="b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0"
Oct 31 19:51:08.426201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.425064    5006 scope.go:122] "RemoveContainer" containerID="b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0"
Oct 31 19:51:08.426201 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:51:08.425508    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0\": container with ID starting with b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0 not found: ID does not exist" containerID="b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0"
Oct 31 19:51:08.426201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.425560    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0"} err="failed to get container status \"b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0\": rpc error: code = NotFound desc = could not find container \"b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0\": container with ID starting with b04aa4675d77474f56ec011d2fac990c7f24b30bd27d1b79b4f2d7079ed700a0 not found: ID does not exist"
Oct 31 19:51:08.574569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.574411    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"]
Oct 31 19:51:08.669798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:08.669250    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31a1ba53-857d86b95-29nx8"]
Oct 31 19:51:09.626681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:09.615643    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a429faf1-b779-4be2-a8a2-ad6a71558cc0" path="/var/lib/kubelet/pods/a429faf1-b779-4be2-a8a2-ad6a71558cc0/volumes"
Oct 31 19:51:10.454571 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:10.454539    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:15.458231 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:15.458175    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:19.699384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:19.699344    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:21.339865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:21.339834    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:21.634730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:21.626007    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:21.846030 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:21.845993    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:21.876389 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:21.876355    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"]
Oct 31 19:51:21.931050 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:21.930953    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9be7bd13-x9svp"]
Oct 31 19:51:22.019566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:22.019536    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:51:22.027212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:22.026637    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:51:22.102773 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:22.097447    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:22.719355 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:22.719313    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:22.954451 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:22.954414    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:23.056531 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:23.056493    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"]
Oct 31 19:51:23.106688 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:23.106640    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc999293-nqdhj"]
Oct 31 19:51:23.186075 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:23.186026    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:23.469356 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:23.469316    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:23.756970 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:23.756874    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:23.798627 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:23.798590    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1adc564-79bd-44d2-9700-0f0ddd9a3e90" path="/var/lib/kubelet/pods/d1adc564-79bd-44d2-9700-0f0ddd9a3e90/volumes"
Oct 31 19:51:23.799753 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:23.799726    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5389803-9527-4b79-acbe-da167c749990" path="/var/lib/kubelet/pods/e5389803-9527-4b79-acbe-da167c749990/volumes"
Oct 31 19:51:24.180633 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:24.180594    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:26.497617 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:26.497570    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:27.904311 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:27.904271    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:28.131212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:28.131176    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:28.621859 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:28.621819    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:29.534360 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:29.534327    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:29.822155 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:29.822063    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:30.008958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:30.008922    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"]
Oct 31 19:51:30.058847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:30.058798    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae778663-rvgqg"]
Oct 31 19:51:30.469109 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:30.469068    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:30.717736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:30.717694    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:31.657827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:31.657792    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:31.659048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:31.658950    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2927278-9741-4e00-adf5-a1457ebd9c18" path="/var/lib/kubelet/pods/c2927278-9741-4e00-adf5-a1457ebd9c18/volumes"
Oct 31 19:51:32.207196 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:32.207149    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:32.825693 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:32.813594    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:33.061007 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:33.060952    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:33.536606 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:33.536568    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:34.705495 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:34.705459    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:51:35.396012 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:51:35.395970    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:51:35.483569 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:35.483528    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:35.769937 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:35.769838    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:35.993290 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:35.993254    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:36.054036 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:36.053947    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"]
Oct 31 19:51:36.121155 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:36.121082    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31a1ba53-j7j69"]
Oct 31 19:51:36.235574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:36.235531    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:37.169559 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:37.169510    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:37.391957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:37.391925    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:37.600792 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:37.600756    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a110888-e4b9-4b17-a71a-72187634d3b4" path="/var/lib/kubelet/pods/9a110888-e4b9-4b17-a71a-72187634d3b4/volumes"
Oct 31 19:51:37.771295 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:37.771262    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:37.994962 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:37.994877    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:38.218426 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:38.218387    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:39.050298 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:39.050262    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"]
Oct 31 19:51:39.100207 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:39.100166    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8434b597-qbb8k"]
Oct 31 19:51:39.601583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:39.601549    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd4a721-1d07-4564-989e-c2b6a52e4fee" path="/var/lib/kubelet/pods/9dd4a721-1d07-4564-989e-c2b6a52e4fee/volumes"
Oct 31 19:51:41.049031 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:41.048996    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:41.264000 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:41.263962    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:41.623643 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:41.623576    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:46.667587 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:46.667552    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:51.638543 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:51.638509    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:55.729135 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.729097    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"]
Oct 31 19:51:55.729837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.729820    5006 cpu_manager.go:414] "containerMap: removing container" podUID="9a110888-e4b9-4b17-a71a-72187634d3b4" containerName="revoke-keys"
Oct 31 19:51:55.729837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.729837    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a110888-e4b9-4b17-a71a-72187634d3b4" containerName="revoke-keys"
Oct 31 19:51:55.729946 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.729859    5006 cpu_manager.go:414] "containerMap: removing container" podUID="6141a7b4-2875-4dad-90b8-2576b934b347" containerName="payload-pre-processing"
Oct 31 19:51:55.729946 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.729864    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6141a7b4-2875-4dad-90b8-2576b934b347" containerName="payload-pre-processing"
Oct 31 19:51:55.729946 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.729887    5006 cpu_manager.go:414] "containerMap: removing container" podUID="a429faf1-b779-4be2-a8a2-ad6a71558cc0" containerName="payload-pre-processing"
Oct 31 19:51:55.729946 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.729891    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a429faf1-b779-4be2-a8a2-ad6a71558cc0" containerName="payload-pre-processing"
Oct 31 19:51:55.730075 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.729996    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a429faf1-b779-4be2-a8a2-ad6a71558cc0" containerName="payload-pre-processing"
Oct 31 19:51:55.730075 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.730011    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a110888-e4b9-4b17-a71a-72187634d3b4" containerName="revoke-keys"
Oct 31 19:51:55.730075 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.730018    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6141a7b4-2875-4dad-90b8-2576b934b347" containerName="payload-pre-processing"
Oct 31 19:51:55.733860 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.733838    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:55.792576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.792540    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"]
Oct 31 19:51:55.832039 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.832000    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:51:55.832039 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.832016    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:51:55.832289 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.832084    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:51:55.840513 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.840481    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:51:55.860360 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.860314    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4xs\" (UniqueName: \"kubernetes.io/projected/4dd7871e-1471-4a10-a826-3e3839e08070-kube-api-access-kh4xs\") pod \"maas-api-revoke-sub-d742a2a7-zll46\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:55.860545 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.860485    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d742a2a7-zll46\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:55.860592 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.860579    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d742a2a7-zll46\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:55.962136 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.962051    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4xs\" (UniqueName: \"kubernetes.io/projected/4dd7871e-1471-4a10-a826-3e3839e08070-kube-api-access-kh4xs\") pod \"maas-api-revoke-sub-d742a2a7-zll46\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:55.962391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.962176    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d742a2a7-zll46\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:55.962391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.962249    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d742a2a7-zll46\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:55.963113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.963085    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d742a2a7-zll46\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:55.964990 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:55.964959    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d742a2a7-zll46\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:56.020401 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:56.020326    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4xs\" (UniqueName: \"kubernetes.io/projected/4dd7871e-1471-4a10-a826-3e3839e08070-kube-api-access-kh4xs\") pod \"maas-api-revoke-sub-d742a2a7-zll46\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:56.049929 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:56.049897    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:56.469759 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:56.469502    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"]
Oct 31 19:51:56.471947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:56.471880    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:56.472197 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:51:56.472169    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd7871e_1471_4a10_a826_3e3839e08070.slice/crio-04d538bf44d303ca229a19356df83ff312ec1d3ba8126bef1d5300f3dd6f7e4b WatchSource:0}: Error finding container 04d538bf44d303ca229a19356df83ff312ec1d3ba8126bef1d5300f3dd6f7e4b: Status 404 returned error can't find the container with id 04d538bf44d303ca229a19356df83ff312ec1d3ba8126bef1d5300f3dd6f7e4b
Oct 31 19:51:57.147302 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:57.147257    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46" event={"ID":"4dd7871e-1471-4a10-a826-3e3839e08070","Type":"ContainerStarted","Data":"60527bf65342a09d780597f9d9b4e1aa62175390006fb6763487eb0eac8cca20"}
Oct 31 19:51:57.147768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:57.147326    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46" event={"ID":"4dd7871e-1471-4a10-a826-3e3839e08070","Type":"ContainerStarted","Data":"04d538bf44d303ca229a19356df83ff312ec1d3ba8126bef1d5300f3dd6f7e4b"}
Oct 31 19:51:57.265695 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:57.265617    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46" podStartSLOduration=2.265600264 podStartE2EDuration="2.265600264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:57.261155232 +0000 UTC m=+2135.790689286" watchObservedRunningTime="2026-10-31 19:51:57.265600264 +0000 UTC m=+2135.795134316"
Oct 31 19:51:58.158618 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:58.158572    5006 generic.go:356] "Generic (PLEG): container finished" podID="4dd7871e-1471-4a10-a826-3e3839e08070" containerID="60527bf65342a09d780597f9d9b4e1aa62175390006fb6763487eb0eac8cca20" exitCode=0
Oct 31 19:51:58.159138 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:58.158672    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46" event={"ID":"4dd7871e-1471-4a10-a826-3e3839e08070","Type":"ContainerDied","Data":"60527bf65342a09d780597f9d9b4e1aa62175390006fb6763487eb0eac8cca20"}
Oct 31 19:51:59.384123 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:59.384086    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:51:59.532455 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:59.532340    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-openshift-ca\") pod \"4dd7871e-1471-4a10-a826-3e3839e08070\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") "
Oct 31 19:51:59.532640 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:59.532475    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd7871e-1471-4a10-a826-3e3839e08070-kube-api-access-kh4xs\" (UniqueName: \"kubernetes.io/projected/4dd7871e-1471-4a10-a826-3e3839e08070-kube-api-access-kh4xs\") pod \"4dd7871e-1471-4a10-a826-3e3839e08070\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") "
Oct 31 19:51:59.532640 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:59.532557    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-xks-ca\") pod \"4dd7871e-1471-4a10-a826-3e3839e08070\" (UID: \"4dd7871e-1471-4a10-a826-3e3839e08070\") "
Oct 31 19:51:59.532880 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:59.532823    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-openshift-ca" pod "4dd7871e-1471-4a10-a826-3e3839e08070" (UID: "4dd7871e-1471-4a10-a826-3e3839e08070"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:59.533460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:59.533433    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:59.536661 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:59.535807    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd7871e-1471-4a10-a826-3e3839e08070-kube-api-access-kh4xs" pod "4dd7871e-1471-4a10-a826-3e3839e08070" (UID: "4dd7871e-1471-4a10-a826-3e3839e08070"). InnerVolumeSpecName "kube-api-access-kh4xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:59.536863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:59.536824    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-xks-ca" pod "4dd7871e-1471-4a10-a826-3e3839e08070" (UID: "4dd7871e-1471-4a10-a826-3e3839e08070"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:51:59.634873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:59.634836    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh4xs\" (UniqueName: \"kubernetes.io/projected/4dd7871e-1471-4a10-a826-3e3839e08070-kube-api-access-kh4xs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:59.634873 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:51:59.634869    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd7871e-1471-4a10-a826-3e3839e08070-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:00.180208 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:00.180174    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"
Oct 31 19:52:00.180378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:00.180213    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46" event={"ID":"4dd7871e-1471-4a10-a826-3e3839e08070","Type":"ContainerDied","Data":"04d538bf44d303ca229a19356df83ff312ec1d3ba8126bef1d5300f3dd6f7e4b"}
Oct 31 19:52:00.180378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:00.180271    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d538bf44d303ca229a19356df83ff312ec1d3ba8126bef1d5300f3dd6f7e4b"
Oct 31 19:52:01.483932 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:01.483861    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:02.051439 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:02.051398    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"]
Oct 31 19:52:02.101438 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:02.101400    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7192868-mvb7x"]
Oct 31 19:52:03.054106 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:03.054068    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"]
Oct 31 19:52:03.104267 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:03.104227    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64972103-t8794"]
Oct 31 19:52:03.600833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:03.600789    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07de6229-b7c1-47ec-af5a-14df75111488" path="/var/lib/kubelet/pods/07de6229-b7c1-47ec-af5a-14df75111488/volumes"
Oct 31 19:52:03.601347 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:03.601327    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafcb30b-2516-46e0-a82b-9069cbb334b6" path="/var/lib/kubelet/pods/dafcb30b-2516-46e0-a82b-9069cbb334b6/volumes"
Oct 31 19:52:04.722130 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:04.722087    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:05.395860 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:52:05.395825    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:52:06.596051 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:06.596002    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:06.837850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:06.837810    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:07.067767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:07.067643    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:10.784498 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:10.784449    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:11.570821 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:11.570789    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:11.805417 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:11.805381    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:13.248723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:13.248608    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:13.933224 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:13.933189    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:14.053784 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:14.053734    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"]
Oct 31 19:52:14.061626 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:14.061588    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeaf9b6d-hkctt"]
Oct 31 19:52:15.600584 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:15.600547    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba86388-27ea-4e3a-8a4a-4556554dea7c" path="/var/lib/kubelet/pods/eba86388-27ea-4e3a-8a4a-4556554dea7c/volumes"
Oct 31 19:52:16.541796 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:16.541728    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:19.098194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:19.098146    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"]
Oct 31 19:52:19.103073 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:19.103037    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb77f644-4thbk"]
Oct 31 19:52:19.602920 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:19.602884    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2287821a-df1a-4443-bbcc-207b2291fbeb" path="/var/lib/kubelet/pods/2287821a-df1a-4443-bbcc-207b2291fbeb/volumes"
Oct 31 19:52:20.122996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:20.122960    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:20.806460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:20.806403    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:21.041591 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:21.041548    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:21.884188 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:21.884125    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:22.124183 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:22.124148    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:22.416086 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:22.416047    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:22.619694 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:22.619612    5006 scope.go:122] "RemoveContainer" containerID="da9f0b31650c2a1387871acd91c7ee873dec3682c2142e23944d9fab0af7fc1e"
Oct 31 19:52:22.636189 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:22.636161    5006 scope.go:122] "RemoveContainer" containerID="ae4bc783e5d9eac988640a2b2dbc091c1d7dd5bc92acee32af6e76f07e5787bb"
Oct 31 19:52:22.647830 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:22.647798    5006 scope.go:122] "RemoveContainer" containerID="4a304b94db2c76c3bdee2bfc32ff51271a5e61a15f7d9c1ffbf8c47910255d57"
Oct 31 19:52:23.874884 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:23.874849    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:25.520066 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:25.520032    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:25.799291 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:25.799174    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:27.817956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:27.817918    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:30.737723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:30.737679    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:31.045951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:31.045818    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:33.659170 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:33.659132    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:35.396604 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:52:35.396565    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:52:38.786196 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:38.786141    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:39.052402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:39.052304    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"]
Oct 31 19:52:39.101870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:39.101833    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763de9ca-p6sxv"]
Oct 31 19:52:39.600579 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:39.600504    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a152abe-7297-4cda-9402-f1e48690a136" path="/var/lib/kubelet/pods/9a152abe-7297-4cda-9402-f1e48690a136/volumes"
Oct 31 19:52:40.302703 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:40.293210    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:52:41.056269 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:41.056224    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"]
Oct 31 19:52:41.103969 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:41.103922    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb98d45-jk2nl"]
Oct 31 19:52:41.627883 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:41.627843    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22f7cee-3ce0-4387-ac41-318b473c6794" path="/var/lib/kubelet/pods/b22f7cee-3ce0-4387-ac41-318b473c6794/volumes"
Oct 31 19:52:42.056722 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:42.055775    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"]
Oct 31 19:52:42.106173 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:42.106127    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c89baa59-fv29q"]
Oct 31 19:52:43.790763 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:43.790693    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ddacb46-c32c-421c-8f7d-6f44adfba25d" path="/var/lib/kubelet/pods/7ddacb46-c32c-421c-8f7d-6f44adfba25d/volumes"
Oct 31 19:52:44.562860 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:44.562819    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:45.052713 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:45.052671    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"]
Oct 31 19:52:45.102488 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:45.102447    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-817f0b73-8jqhr"]
Oct 31 19:52:45.605704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:45.605659    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42070ba2-fbe1-4b38-afc6-2cc3003006b1" path="/var/lib/kubelet/pods/42070ba2-fbe1-4b38-afc6-2cc3003006b1/volumes"
Oct 31 19:52:46.049121 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:46.049037    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"]
Oct 31 19:52:46.099029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:46.098993    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3beeba99-xncgm"]
Oct 31 19:52:47.601666 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:47.601608    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271a1724-3bee-4943-9639-90f9cf8233bb" path="/var/lib/kubelet/pods/271a1724-3bee-4943-9639-90f9cf8233bb/volumes"
Oct 31 19:52:48.308464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:48.308417    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:48.976468 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:48.976425    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:49.503869 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:49.503827    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:50.023436 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:50.023384    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:50.056133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:50.056094    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"]
Oct 31 19:52:50.104928 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:50.104889    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a46f39-79p7r"]
Oct 31 19:52:51.601715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:51.601640    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec52aa5f-4450-4549-828a-ac1b3ae29b39" path="/var/lib/kubelet/pods/ec52aa5f-4450-4549-828a-ac1b3ae29b39/volumes"
Oct 31 19:52:51.604136 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:51.604104    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"]
Oct 31 19:52:51.604366 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:51.604343    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy" containerID="cri-o://51b6872224cf2f254b460d1fb025ef9dfcc7bce84f0ba4a5bc3010ce7db519f1" gracePeriod=30
Oct 31 19:52:53.257246 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:53.257210    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:53.540844 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:53.540758    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:55.513779 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:55.513724    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:56.607607 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:56.607566    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:56.880879 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:56.880771    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:56.897662 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:56.897422    5006 generic.go:356] "Generic (PLEG): container finished" podID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerID="51b6872224cf2f254b460d1fb025ef9dfcc7bce84f0ba4a5bc3010ce7db519f1" exitCode=0
Oct 31 19:52:56.897662 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:56.897612    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" event={"ID":"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262","Type":"ContainerDied","Data":"51b6872224cf2f254b460d1fb025ef9dfcc7bce84f0ba4a5bc3010ce7db519f1"}
Oct 31 19:52:57.061221 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.061174    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:52:57.189965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.189860    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-podinfo\") pod \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") "
Oct 31 19:52:57.190160 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.190078    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-envoy\") pod \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") "
Oct 31 19:52:57.190235 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.190163    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-kube-api-access-sbk8f\" (UniqueName: \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-kube-api-access-sbk8f\") pod \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") "
Oct 31 19:52:57.190326 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.190310    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-data\") pod \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") "
Oct 31 19:52:57.190388 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.190367    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-credential-socket\") pod \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") "
Oct 31 19:52:57.190444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.190402    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-token\" (UniqueName: \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-token\") pod \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") "
Oct 31 19:52:57.190576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.190456    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-socket\") pod \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") "
Oct 31 19:52:57.190576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.190518    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istiod-ca-cert\") pod \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") "
Oct 31 19:52:57.190576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.190553    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-certs\") pod \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\" (UID: \"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262\") "
Oct 31 19:52:57.190767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.190610    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-data" pod "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" (UID: "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:57.190935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.190901    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-socket" pod "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" (UID: "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:57.191157 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.191125    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istiod-ca-cert" pod "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" (UID: "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:52:57.191359 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.191332    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:57.191478 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.191361    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:57.191478 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.191376    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:57.191810 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.191784    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-credential-socket" pod "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" (UID: "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:57.191945 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.191925    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-certs" pod "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" (UID: "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:57.193067 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.193044    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-envoy" pod "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" (UID: "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:57.193357 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.193331    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-token" pod "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" (UID: "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:57.193501 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.193482    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-kube-api-access-sbk8f" pod "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" (UID: "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262"). InnerVolumeSpecName "kube-api-access-sbk8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:57.196134 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.196093    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-podinfo" pod "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" (UID: "aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:52:57.296711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.296666    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:57.296711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.296703    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:57.296711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.296716    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:57.297024 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.296728    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:57.297024 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.296741    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:57.297024 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.296754    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbk8f\" (UniqueName: \"kubernetes.io/projected/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262-kube-api-access-sbk8f\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:57.907903 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.907812    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"
Oct 31 19:52:57.907903 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.907842    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm" event={"ID":"aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262","Type":"ContainerDied","Data":"eb1b4921f31c8d4b11c87de94aaa02dc6edf073db9d5ab811899a612013b05aa"}
Oct 31 19:52:57.908449 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:57.907918    5006 scope.go:122] "RemoveContainer" containerID="51b6872224cf2f254b460d1fb025ef9dfcc7bce84f0ba4a5bc3010ce7db519f1"
Oct 31 19:52:58.125442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:58.125405    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"]
Oct 31 19:52:58.176200 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:58.176118    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-tnrxm"]
Oct 31 19:52:59.052414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:59.052326    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"]
Oct 31 19:52:59.102736 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:59.102693    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3baecb-qrp59"]
Oct 31 19:52:59.599635 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:59.599599    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9" path="/var/lib/kubelet/pods/a54fdb88-3a3a-43fb-9579-ccc39eb5b6a9/volumes"
Oct 31 19:52:59.600203 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:52:59.600188    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" path="/var/lib/kubelet/pods/aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262/volumes"
Oct 31 19:53:01.227100 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:01.226997    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:05.399264 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:53:05.399224    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:53:06.051865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:06.051827    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"]
Oct 31 19:53:06.062082 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:06.062050    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cc06507-zxkt7"]
Oct 31 19:53:06.552452 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:06.552389    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:07.600178 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:07.600138    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a6803fa-8a7e-44b1-b2d1-ed9d6179f939" path="/var/lib/kubelet/pods/0a6803fa-8a7e-44b1-b2d1-ed9d6179f939/volumes"
Oct 31 19:53:11.050162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:11.050121    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"]
Oct 31 19:53:11.100259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:11.100224    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"]
Oct 31 19:53:11.104862 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:11.104834    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e348152-v4xpg"]
Oct 31 19:53:11.108956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:11.108923    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa4c76c9-xcb24"]
Oct 31 19:53:11.528112 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:11.528081    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:11.602568 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:11.602531    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777f4f51-efa9-4be1-a86f-626692778040" path="/var/lib/kubelet/pods/777f4f51-efa9-4be1-a86f-626692778040/volumes"
Oct 31 19:53:11.603192 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:11.603173    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e14978-4d67-42e1-93a6-268a6deedd69" path="/var/lib/kubelet/pods/99e14978-4d67-42e1-93a6-268a6deedd69/volumes"
Oct 31 19:53:12.051955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:12.051920    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"]
Oct 31 19:53:12.100871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:12.100825    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"]
Oct 31 19:53:12.104999 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:12.104968    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e29362c-pkrvd"]
Oct 31 19:53:12.108818 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:12.108789    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2adf7fb-lmx77"]
Oct 31 19:53:13.600666 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:13.600623    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="196e332b-96d9-4f71-98ee-3ca1da1d5eb5" path="/var/lib/kubelet/pods/196e332b-96d9-4f71-98ee-3ca1da1d5eb5/volumes"
Oct 31 19:53:13.601234 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:13.601210    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f9d736-7844-4ddc-861e-007871cfa749" path="/var/lib/kubelet/pods/a5f9d736-7844-4ddc-861e-007871cfa749/volumes"
Oct 31 19:53:15.015907 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.015869    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"]
Oct 31 19:53:15.016604 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.016587    5006 cpu_manager.go:414] "containerMap: removing container" podUID="4dd7871e-1471-4a10-a826-3e3839e08070" containerName="revoke-keys"
Oct 31 19:53:15.016604 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.016605    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd7871e-1471-4a10-a826-3e3839e08070" containerName="revoke-keys"
Oct 31 19:53:15.016704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.016645    5006 cpu_manager.go:414] "containerMap: removing container" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy"
Oct 31 19:53:15.016704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.016666    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy"
Oct 31 19:53:15.016810 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.016798    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd7871e-1471-4a10-a826-3e3839e08070" containerName="revoke-keys"
Oct 31 19:53:15.016860 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.016819    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1b581a-dd6b-4f04-b87d-dc4e5fbe6262" containerName="istio-proxy"
Oct 31 19:53:15.020546 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.020526    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:15.076572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.076520    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"]
Oct 31 19:53:15.121239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.121199    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:53:15.121239 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.121228    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:53:15.121533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.121324    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:53:15.121533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.121413    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:53:15.178671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.178616    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5jt\" (UniqueName: \"kubernetes.io/projected/4000f9e9-71e6-4053-8077-e4c19f856946-kube-api-access-hw5jt\") pod \"maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:15.178889 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.178828    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:15.179061 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.179035    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:15.280576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.280469    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:15.280812 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.280623    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5jt\" (UniqueName: \"kubernetes.io/projected/4000f9e9-71e6-4053-8077-e4c19f856946-kube-api-access-hw5jt\") pod \"maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:15.280812 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.280772    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:15.281428 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.281401    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:15.283089 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.283067    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:15.338427 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.338389    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5jt\" (UniqueName: \"kubernetes.io/projected/4000f9e9-71e6-4053-8077-e4c19f856946-kube-api-access-hw5jt\") pod \"maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:15.637228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:15.637191    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:16.012863 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:53:16.012826    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4000f9e9_71e6_4053_8077_e4c19f856946.slice/crio-408abb302468106bfadb92c83448b1f7e7b02d4ac6212f54d986f8803fee3444 WatchSource:0}: Error finding container 408abb302468106bfadb92c83448b1f7e7b02d4ac6212f54d986f8803fee3444: Status 404 returned error can't find the container with id 408abb302468106bfadb92c83448b1f7e7b02d4ac6212f54d986f8803fee3444
Oct 31 19:53:16.018541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:16.018512    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"]
Oct 31 19:53:16.065046 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:16.065009    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr" event={"ID":"4000f9e9-71e6-4053-8077-e4c19f856946","Type":"ContainerStarted","Data":"408abb302468106bfadb92c83448b1f7e7b02d4ac6212f54d986f8803fee3444"}
Oct 31 19:53:16.840935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:16.840901    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:17.081853 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:17.081806    5006 generic.go:356] "Generic (PLEG): container finished" podID="4000f9e9-71e6-4053-8077-e4c19f856946" containerID="6f49facf0b5a54b32351938ce28d69e8f780ae1f0131bf5a20b5519a6d190b77" exitCode=0
Oct 31 19:53:17.082508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:17.082158    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr" event={"ID":"4000f9e9-71e6-4053-8077-e4c19f856946","Type":"ContainerDied","Data":"6f49facf0b5a54b32351938ce28d69e8f780ae1f0131bf5a20b5519a6d190b77"}
Oct 31 19:53:18.262330 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:18.262295    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:18.343236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:18.343146    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-xks-ca\") pod \"4000f9e9-71e6-4053-8077-e4c19f856946\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") "
Oct 31 19:53:18.343236 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:18.343198    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4000f9e9-71e6-4053-8077-e4c19f856946-kube-api-access-hw5jt\" (UniqueName: \"kubernetes.io/projected/4000f9e9-71e6-4053-8077-e4c19f856946-kube-api-access-hw5jt\") pod \"4000f9e9-71e6-4053-8077-e4c19f856946\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") "
Oct 31 19:53:18.343418 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:18.343300    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-openshift-ca\") pod \"4000f9e9-71e6-4053-8077-e4c19f856946\" (UID: \"4000f9e9-71e6-4053-8077-e4c19f856946\") "
Oct 31 19:53:18.343733 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:18.343705    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-openshift-ca" pod "4000f9e9-71e6-4053-8077-e4c19f856946" (UID: "4000f9e9-71e6-4053-8077-e4c19f856946"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:53:18.344083 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:18.344063    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:18.345416 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:18.345389    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-xks-ca" pod "4000f9e9-71e6-4053-8077-e4c19f856946" (UID: "4000f9e9-71e6-4053-8077-e4c19f856946"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:53:18.345549 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:18.345511    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4000f9e9-71e6-4053-8077-e4c19f856946-kube-api-access-hw5jt" pod "4000f9e9-71e6-4053-8077-e4c19f856946" (UID: "4000f9e9-71e6-4053-8077-e4c19f856946"). InnerVolumeSpecName "kube-api-access-hw5jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:18.445588 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:18.445549    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4000f9e9-71e6-4053-8077-e4c19f856946-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:18.445588 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:18.445585    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw5jt\" (UniqueName: \"kubernetes.io/projected/4000f9e9-71e6-4053-8077-e4c19f856946-kube-api-access-hw5jt\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:19.100521 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:19.100493    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"
Oct 31 19:53:19.100722 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:19.100535    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr" event={"ID":"4000f9e9-71e6-4053-8077-e4c19f856946","Type":"ContainerDied","Data":"408abb302468106bfadb92c83448b1f7e7b02d4ac6212f54d986f8803fee3444"}
Oct 31 19:53:19.100722 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:19.100583    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="408abb302468106bfadb92c83448b1f7e7b02d4ac6212f54d986f8803fee3444"
Oct 31 19:53:21.054210 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:21.054171    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"]
Oct 31 19:53:21.105396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:21.105351    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc83de9b-nsd42"]
Oct 31 19:53:21.600942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:21.600908    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d1d2b2-736d-4566-8851-7d8e296aeb22" path="/var/lib/kubelet/pods/80d1d2b2-736d-4566-8851-7d8e296aeb22/volumes"
Oct 31 19:53:22.221492 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.221454    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:22.592660 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.592614    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"]
Oct 31 19:53:22.592848 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.592824    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" containerID="cri-o://d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6" gracePeriod=30
Oct 31 19:53:22.645762 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.645721    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"]
Oct 31 19:53:22.646013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.645978    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy" containerID="cri-o://0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6" gracePeriod=30
Oct 31 19:53:22.710640 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.710608    5006 scope.go:122] "RemoveContainer" containerID="b98d375fe76309b70b9b56020d28c680e416f0b45160f63853edcfcd8a0cbe58"
Oct 31 19:53:22.728330 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.728306    5006 scope.go:122] "RemoveContainer" containerID="14ce114c50e6ec01edde8b35f4f28e0e634cbea1af45431c2d6b8b6bb5a7043f"
Oct 31 19:53:22.731743 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.731711    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:22.739594 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.739570    5006 scope.go:122] "RemoveContainer" containerID="a02e1437c2f4eb778af8cfd45b1c9289a9c997fa2f8b8744ff17547918c34218"
Oct 31 19:53:22.752364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.752340    5006 scope.go:122] "RemoveContainer" containerID="0548e306f6d9d9ddf506baebd37b9990c1410af8db9c6ccf18e6aa42fd631688"
Oct 31 19:53:22.764096 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:22.764071    5006 scope.go:122] "RemoveContainer" containerID="f1c244bf4cd2d116c2eb1c7a9bac5ac8151370f23767a0085ccea48ee9431d63"
Oct 31 19:53:23.191071 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:23.191025    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:53:23.191255 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:23.191097    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:53:24.152422 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:24.152384    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:25.248921 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.248881    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"]
Oct 31 19:53:25.249586 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.249569    5006 cpu_manager.go:414] "containerMap: removing container" podUID="4000f9e9-71e6-4053-8077-e4c19f856946" containerName="revoke-keys"
Oct 31 19:53:25.249586 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.249586    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4000f9e9-71e6-4053-8077-e4c19f856946" containerName="revoke-keys"
Oct 31 19:53:25.249745 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.249735    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4000f9e9-71e6-4053-8077-e4c19f856946" containerName="revoke-keys"
Oct 31 19:53:25.255360 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.255335    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.311248 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.311215    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"]
Oct 31 19:53:25.356132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.356097    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:53:25.356392 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.356196    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:53:25.358107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.358077    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2065539-2931-45f4-8110-dcb75776f711-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.358267 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.358239    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46p7\" (UniqueName: \"kubernetes.io/projected/f2065539-2931-45f4-8110-dcb75776f711-kube-api-access-v46p7\") pod \"maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.358539 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.358517    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2065539-2931-45f4-8110-dcb75776f711-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.413711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.411248    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:53:25.413711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.411375    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:53:25.459639 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.459589    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2065539-2931-45f4-8110-dcb75776f711-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.459889 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.459803    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2065539-2931-45f4-8110-dcb75776f711-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.459964 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.459895    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46p7\" (UniqueName: \"kubernetes.io/projected/f2065539-2931-45f4-8110-dcb75776f711-kube-api-access-v46p7\") pod \"maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.460442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.460417    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2065539-2931-45f4-8110-dcb75776f711-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.462417 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.462392    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2065539-2931-45f4-8110-dcb75776f711-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.518997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.518916    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46p7\" (UniqueName: \"kubernetes.io/projected/f2065539-2931-45f4-8110-dcb75776f711-kube-api-access-v46p7\") pod \"maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.575997 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.575956    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:25.983082 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:53:25.983047    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2065539_2931_45f4_8110_dcb75776f711.slice/crio-2682c5691bbc46f54ff7af1313cf81aa7ba2f4a03d6f42ef9151fcc2537854cb WatchSource:0}: Error finding container 2682c5691bbc46f54ff7af1313cf81aa7ba2f4a03d6f42ef9151fcc2537854cb: Status 404 returned error can't find the container with id 2682c5691bbc46f54ff7af1313cf81aa7ba2f4a03d6f42ef9151fcc2537854cb
Oct 31 19:53:25.987260 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:25.987230    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"]
Oct 31 19:53:26.161905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:26.161853    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8" event={"ID":"f2065539-2931-45f4-8110-dcb75776f711","Type":"ContainerStarted","Data":"2682c5691bbc46f54ff7af1313cf81aa7ba2f4a03d6f42ef9151fcc2537854cb"}
Oct 31 19:53:27.171552 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:27.171514    5006 generic.go:356] "Generic (PLEG): container finished" podID="f2065539-2931-45f4-8110-dcb75776f711" containerID="00bebbbdea78ebe0ed1e3d2f18101aec680fb36236566f86a60fd928b795b085" exitCode=0
Oct 31 19:53:27.172021 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:27.171573    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8" event={"ID":"f2065539-2931-45f4-8110-dcb75776f711","Type":"ContainerDied","Data":"00bebbbdea78ebe0ed1e3d2f18101aec680fb36236566f86a60fd928b795b085"}
Oct 31 19:53:27.816138 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:27.815637    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:27.836100 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:27.834138    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 19:53:27.836100 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:27.834212    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 19:53:28.042713 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.042677    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:28.107863 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.107825    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:53:28.113000 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.112969    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:53:28.182249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.182120    5006 generic.go:356] "Generic (PLEG): container finished" podID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerID="0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6" exitCode=0
Oct 31 19:53:28.182249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.182184    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"
Oct 31 19:53:28.182249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.182190    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" event={"ID":"23d6fc0d-24c8-4770-8a37-101c0775b924","Type":"ContainerDied","Data":"0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6"}
Oct 31 19:53:28.182892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.182332    5006 scope.go:122] "RemoveContainer" containerID="0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6"
Oct 31 19:53:28.182892 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.182680    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr" event={"ID":"23d6fc0d-24c8-4770-8a37-101c0775b924","Type":"ContainerDied","Data":"d66e5dd92727228114ebabae4f163e136a599562e68c0077661b913660c04d3e"}
Oct 31 19:53:28.184940 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.184916    5006 generic.go:356] "Generic (PLEG): container finished" podID="2f72d787-e063-405e-b385-636791c04833" containerID="d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6" exitCode=0
Oct 31 19:53:28.185064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.184985    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"
Oct 31 19:53:28.185064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.185018    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" event={"ID":"2f72d787-e063-405e-b385-636791c04833","Type":"ContainerDied","Data":"d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6"}
Oct 31 19:53:28.185064 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.185059    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv" event={"ID":"2f72d787-e063-405e-b385-636791c04833","Type":"ContainerDied","Data":"3cdb7cec925b97a04572798af5f3650c933c99e9061a90b885b7495fce8fafc6"}
Oct 31 19:53:28.199734 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.199706    5006 scope.go:122] "RemoveContainer" containerID="0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6"
Oct 31 19:53:28.200154 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:53:28.200125    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6\": container with ID starting with 0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6 not found: ID does not exist" containerID="0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6"
Oct 31 19:53:28.200245 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.200167    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6"} err="failed to get container status \"0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6\": rpc error: code = NotFound desc = could not find container \"0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6\": container with ID starting with 0c74cadf0dec521823b4ae176c680fd364f7115355a532a13712c61f3f5ceca6 not found: ID does not exist"
Oct 31 19:53:28.200245 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.200191    5006 scope.go:122] "RemoveContainer" containerID="d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6"
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219167    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-podinfo\") pod \"23d6fc0d-24c8-4770-8a37-101c0775b924\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") "
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219228    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-credential-socket\") pod \"2f72d787-e063-405e-b385-636791c04833\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") "
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219309    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-istio-token\" (UniqueName: \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-istio-token\") pod \"2f72d787-e063-405e-b385-636791c04833\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") "
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219380    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-kube-api-access-hmlx7\" (UniqueName: \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-kube-api-access-hmlx7\") pod \"23d6fc0d-24c8-4770-8a37-101c0775b924\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") "
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219413    5006 scope.go:122] "RemoveContainer" containerID="d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6"
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219440    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-kube-api-access-ffsz6\" (UniqueName: \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-kube-api-access-ffsz6\") pod \"2f72d787-e063-405e-b385-636791c04833\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") "
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219468    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f72d787-e063-405e-b385-636791c04833-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f72d787-e063-405e-b385-636791c04833-istiod-ca-cert\") pod \"2f72d787-e063-405e-b385-636791c04833\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") "
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219510    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d6fc0d-24c8-4770-8a37-101c0775b924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d6fc0d-24c8-4770-8a37-101c0775b924-istiod-ca-cert\") pod \"23d6fc0d-24c8-4770-8a37-101c0775b924\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") "
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219562    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f72d787-e063-405e-b385-636791c04833-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f72d787-e063-405e-b385-636791c04833-istio-podinfo\") pod \"2f72d787-e063-405e-b385-636791c04833\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") "
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219612    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-certs\") pod \"2f72d787-e063-405e-b385-636791c04833\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") "
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219688    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-token\" (UniqueName: \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-token\") pod \"23d6fc0d-24c8-4770-8a37-101c0775b924\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") "
Oct 31 19:53:28.219776 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219723    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-data\") pod \"23d6fc0d-24c8-4770-8a37-101c0775b924\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") "
Oct 31 19:53:28.220500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219822    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-envoy\") pod \"23d6fc0d-24c8-4770-8a37-101c0775b924\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") "
Oct 31 19:53:28.220500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219929    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-socket\") pod \"23d6fc0d-24c8-4770-8a37-101c0775b924\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") "
Oct 31 19:53:28.220500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219965    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-certs\") pod \"23d6fc0d-24c8-4770-8a37-101c0775b924\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") "
Oct 31 19:53:28.220500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.219978    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-credential-socket" pod "2f72d787-e063-405e-b385-636791c04833" (UID: "2f72d787-e063-405e-b385-636791c04833"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:28.220500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.220049    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-data\") pod \"2f72d787-e063-405e-b385-636791c04833\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") "
Oct 31 19:53:28.220500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.220130    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-envoy\") pod \"2f72d787-e063-405e-b385-636791c04833\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") "
Oct 31 19:53:28.220500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.220176    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-socket\") pod \"2f72d787-e063-405e-b385-636791c04833\" (UID: \"2f72d787-e063-405e-b385-636791c04833\") "
Oct 31 19:53:28.220500 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.220238    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-credential-socket\") pod \"23d6fc0d-24c8-4770-8a37-101c0775b924\" (UID: \"23d6fc0d-24c8-4770-8a37-101c0775b924\") "
Oct 31 19:53:28.220965 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:53:28.220605    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6\": container with ID starting with d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6 not found: ID does not exist" containerID="d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6"
Oct 31 19:53:28.220965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.220642    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6"} err="failed to get container status \"d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6\": rpc error: code = NotFound desc = could not find container \"d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6\": container with ID starting with d113beefcdaf1c82efc71135f57c6bc213abfb914b0a49997098224f3f0d57e6 not found: ID does not exist"
Oct 31 19:53:28.220965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.220043    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-certs" pod "2f72d787-e063-405e-b385-636791c04833" (UID: "2f72d787-e063-405e-b385-636791c04833"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:28.220965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.220487    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-data" pod "2f72d787-e063-405e-b385-636791c04833" (UID: "2f72d787-e063-405e-b385-636791c04833"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:28.220965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.220909    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f72d787-e063-405e-b385-636791c04833-istiod-ca-cert" pod "2f72d787-e063-405e-b385-636791c04833" (UID: "2f72d787-e063-405e-b385-636791c04833"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:53:28.221283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.221077    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.221283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.221100    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f72d787-e063-405e-b385-636791c04833-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.221283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.221114    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.221283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.221129    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.221283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.221241    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-credential-socket" pod "23d6fc0d-24c8-4770-8a37-101c0775b924" (UID: "23d6fc0d-24c8-4770-8a37-101c0775b924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:28.221555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.221306    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-socket" pod "2f72d787-e063-405e-b385-636791c04833" (UID: "2f72d787-e063-405e-b385-636791c04833"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:28.221555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.221325    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d6fc0d-24c8-4770-8a37-101c0775b924-istiod-ca-cert" pod "23d6fc0d-24c8-4770-8a37-101c0775b924" (UID: "23d6fc0d-24c8-4770-8a37-101c0775b924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:53:28.221555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.221460    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-data" pod "23d6fc0d-24c8-4770-8a37-101c0775b924" (UID: "23d6fc0d-24c8-4770-8a37-101c0775b924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:28.221742 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.221593    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-socket" pod "23d6fc0d-24c8-4770-8a37-101c0775b924" (UID: "23d6fc0d-24c8-4770-8a37-101c0775b924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:28.221742 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.221694    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-certs" pod "23d6fc0d-24c8-4770-8a37-101c0775b924" (UID: "23d6fc0d-24c8-4770-8a37-101c0775b924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:28.224788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.222832    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-kube-api-access-ffsz6" pod "2f72d787-e063-405e-b385-636791c04833" (UID: "2f72d787-e063-405e-b385-636791c04833"). InnerVolumeSpecName "kube-api-access-ffsz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:28.224788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.223988    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-kube-api-access-hmlx7" pod "23d6fc0d-24c8-4770-8a37-101c0775b924" (UID: "23d6fc0d-24c8-4770-8a37-101c0775b924"). InnerVolumeSpecName "kube-api-access-hmlx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:28.224788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.224289    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f72d787-e063-405e-b385-636791c04833-istio-podinfo" pod "2f72d787-e063-405e-b385-636791c04833" (UID: "2f72d787-e063-405e-b385-636791c04833"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:53:28.224788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.224314    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-envoy" pod "2f72d787-e063-405e-b385-636791c04833" (UID: "2f72d787-e063-405e-b385-636791c04833"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:28.224788 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.224776    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-token" pod "23d6fc0d-24c8-4770-8a37-101c0775b924" (UID: "23d6fc0d-24c8-4770-8a37-101c0775b924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:28.225285 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.225260    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-istio-token" pod "2f72d787-e063-405e-b385-636791c04833" (UID: "2f72d787-e063-405e-b385-636791c04833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:28.225387 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.225353    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-podinfo" pod "23d6fc0d-24c8-4770-8a37-101c0775b924" (UID: "23d6fc0d-24c8-4770-8a37-101c0775b924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:53:28.225471 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.225457    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-envoy" pod "23d6fc0d-24c8-4770-8a37-101c0775b924" (UID: "23d6fc0d-24c8-4770-8a37-101c0775b924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:28.322424 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322387    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmlx7\" (UniqueName: \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-kube-api-access-hmlx7\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322423    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffsz6\" (UniqueName: \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-kube-api-access-ffsz6\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322458    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d6fc0d-24c8-4770-8a37-101c0775b924-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322472    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f72d787-e063-405e-b385-636791c04833-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322486    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322500    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322513    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322527    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322541    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322554    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322568    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f72d787-e063-405e-b385-636791c04833-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322584    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d6fc0d-24c8-4770-8a37-101c0775b924-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322600    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d6fc0d-24c8-4770-8a37-101c0775b924-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.322610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.322615    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f72d787-e063-405e-b385-636791c04833-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.389724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.389692    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:28.525368 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.525324    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2065539-2931-45f4-8110-dcb75776f711-kube-api-access-v46p7\" (UniqueName: \"kubernetes.io/projected/f2065539-2931-45f4-8110-dcb75776f711-kube-api-access-v46p7\") pod \"f2065539-2931-45f4-8110-dcb75776f711\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") "
Oct 31 19:53:28.525558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.525402    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2065539-2931-45f4-8110-dcb75776f711-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2065539-2931-45f4-8110-dcb75776f711-maas-api-xks-ca\") pod \"f2065539-2931-45f4-8110-dcb75776f711\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") "
Oct 31 19:53:28.525612 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.525570    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2065539-2931-45f4-8110-dcb75776f711-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2065539-2931-45f4-8110-dcb75776f711-maas-api-openshift-ca\") pod \"f2065539-2931-45f4-8110-dcb75776f711\" (UID: \"f2065539-2931-45f4-8110-dcb75776f711\") "
Oct 31 19:53:28.525975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.525950    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2065539-2931-45f4-8110-dcb75776f711-maas-api-openshift-ca" pod "f2065539-2931-45f4-8110-dcb75776f711" (UID: "f2065539-2931-45f4-8110-dcb75776f711"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:53:28.526128 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.526113    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2065539-2931-45f4-8110-dcb75776f711-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.527794 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.527770    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2065539-2931-45f4-8110-dcb75776f711-maas-api-xks-ca" pod "f2065539-2931-45f4-8110-dcb75776f711" (UID: "f2065539-2931-45f4-8110-dcb75776f711"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:53:28.527849 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.527798    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2065539-2931-45f4-8110-dcb75776f711-kube-api-access-v46p7" pod "f2065539-2931-45f4-8110-dcb75776f711" (UID: "f2065539-2931-45f4-8110-dcb75776f711"). InnerVolumeSpecName "kube-api-access-v46p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:28.628103 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.628062    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v46p7\" (UniqueName: \"kubernetes.io/projected/f2065539-2931-45f4-8110-dcb75776f711-kube-api-access-v46p7\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.628103 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.628103    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2065539-2931-45f4-8110-dcb75776f711-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:28.739261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.739219    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"]
Oct 31 19:53:28.753318 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.753242    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-lqkmv"]
Oct 31 19:53:28.918792 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.918724    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"]
Oct 31 19:53:28.929058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:28.928994    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-mm9nr"]
Oct 31 19:53:29.064847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:29.064754    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"]
Oct 31 19:53:29.130431 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:29.130375    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c135fa-xh2h9"]
Oct 31 19:53:29.274466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:29.274426    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8" event={"ID":"f2065539-2931-45f4-8110-dcb75776f711","Type":"ContainerDied","Data":"2682c5691bbc46f54ff7af1313cf81aa7ba2f4a03d6f42ef9151fcc2537854cb"}
Oct 31 19:53:29.275043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:29.274509    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2682c5691bbc46f54ff7af1313cf81aa7ba2f4a03d6f42ef9151fcc2537854cb"
Oct 31 19:53:29.276796 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:29.276766    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"
Oct 31 19:53:29.600081 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:29.600049    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" path="/var/lib/kubelet/pods/23d6fc0d-24c8-4770-8a37-101c0775b924/volumes"
Oct 31 19:53:29.600683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:29.600661    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f72d787-e063-405e-b385-636791c04833" path="/var/lib/kubelet/pods/2f72d787-e063-405e-b385-636791c04833/volumes"
Oct 31 19:53:29.601164 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:29.601146    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982a0aa3-ccc3-4b76-9a99-ba47c45a3f67" path="/var/lib/kubelet/pods/982a0aa3-ccc3-4b76-9a99-ba47c45a3f67/volumes"
Oct 31 19:53:33.185271 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:33.185232    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:35.048636 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.048597    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"]
Oct 31 19:53:35.098686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.098637    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42247df1-6vv7c"]
Oct 31 19:53:35.179017 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.178986    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:35.398407 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:53:35.398366    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:53:35.601074 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.601039    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48207f5c-b0fe-45b1-9114-5cdef66319ef" path="/var/lib/kubelet/pods/48207f5c-b0fe-45b1-9114-5cdef66319ef/volumes"
Oct 31 19:53:35.632474 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.632431    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"]
Oct 31 19:53:35.633207 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.633189    5006 cpu_manager.go:414] "containerMap: removing container" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy"
Oct 31 19:53:35.633268 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.633209    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy"
Oct 31 19:53:35.633268 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.633230    5006 cpu_manager.go:414] "containerMap: removing container" podUID="f2065539-2931-45f4-8110-dcb75776f711" containerName="revoke-keys"
Oct 31 19:53:35.633268 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.633235    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2065539-2931-45f4-8110-dcb75776f711" containerName="revoke-keys"
Oct 31 19:53:35.633364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.633281    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy"
Oct 31 19:53:35.633364 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.633286    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy"
Oct 31 19:53:35.633428 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.633381    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f72d787-e063-405e-b385-636791c04833" containerName="istio-proxy"
Oct 31 19:53:35.633428 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.633393    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d6fc0d-24c8-4770-8a37-101c0775b924" containerName="istio-proxy"
Oct 31 19:53:35.633428 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.633401    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2065539-2931-45f4-8110-dcb75776f711" containerName="revoke-keys"
Oct 31 19:53:35.638456 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.638432    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.668724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.648144    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-socket\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.668724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.648205    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-envoy\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.668724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.648319    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-podinfo\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.668724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.648834    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8nqh\" (UniqueName: \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-kube-api-access-m8nqh\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.668724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.648877    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-data\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.668724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.648923    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-credential-socket\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.668724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.648960    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-certs\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.668724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.649006    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istiod-ca-cert\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.668724 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.649054    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-token\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.755675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.723925    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"]
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.792927    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-podinfo\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.793050    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8nqh\" (UniqueName: \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-kube-api-access-m8nqh\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.793090    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-data\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.793135    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-credential-socket\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.793160    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-certs\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.793208    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istiod-ca-cert\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.793260    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-token\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.793438    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-socket\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.793488    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-envoy\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.797134    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istiod-ca-cert\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.797414    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-certs\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.797703    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-data\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.798151    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-socket\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.799675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.799190    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-credential-socket\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.868716 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.808060    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-podinfo\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.868716 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.858410    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:35.919727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.912725    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-envoy\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.919727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.913264    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-7c0d34-openshift-default-dockercfg-5z97m\""
Oct 31 19:53:35.982566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.979370    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-token\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:35.982566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:35.981073    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8nqh\" (UniqueName: \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-kube-api-access-m8nqh\") pod \"e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") " pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:36.217114 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:36.217024    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:36.262201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:36.262160    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:36.696031 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:53:36.695995    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8c5d8a_bd52_4cfa_ac40_b32b8fbb3d78.slice/crio-159dd88db34226588fc33c9dd850c04ccc891979f65985dd008717fb20656340 WatchSource:0}: Error finding container 159dd88db34226588fc33c9dd850c04ccc891979f65985dd008717fb20656340: Status 404 returned error can't find the container with id 159dd88db34226588fc33c9dd850c04ccc891979f65985dd008717fb20656340
Oct 31 19:53:36.699352 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:36.699304    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:36.699503 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:36.699408    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:36.699503 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:36.699457    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:36.700827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:36.700803    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"]
Oct 31 19:53:36.815757 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:36.815718    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:37.045260 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:37.045176    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:37.361043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:37.361005    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" event={"ID":"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78","Type":"ContainerStarted","Data":"69db270188f1bec2cd5b94539fa5301e1d35483d0ba12cb38451f232fe23eaae"}
Oct 31 19:53:37.361453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:37.361073    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" event={"ID":"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78","Type":"ContainerStarted","Data":"159dd88db34226588fc33c9dd850c04ccc891979f65985dd008717fb20656340"}
Oct 31 19:53:37.532248 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:37.532185    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" podStartSLOduration=2.532168199 podStartE2EDuration="2.532168199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:37.528800005 +0000 UTC m=+2236.058334079" watchObservedRunningTime="2026-10-31 19:53:37.532168199 +0000 UTC m=+2236.061702252"
Oct 31 19:53:37.596013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:37.595974    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:38.070672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:38.068081    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"]
Oct 31 19:53:38.121676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:38.120299    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dadc90d-54xgm"]
Oct 31 19:53:38.267004 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:38.266042    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:38.288291 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:38.288257    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:38.361511 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:38.361454    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:38.395545 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:38.395513    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:38.396935 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:38.396913    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:53:38.826576 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:38.826349    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:39.058737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:39.058700    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:39.600437 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:39.600402    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d96a6f-628f-4030-9127-0e011cc1eeb5" path="/var/lib/kubelet/pods/c1d96a6f-628f-4030-9127-0e011cc1eeb5/volumes"
Oct 31 19:53:40.055301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.055209    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"]
Oct 31 19:53:40.102362 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.102325    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cfbfd03-sgjv6"]
Oct 31 19:53:40.151676 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.150743    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"]
Oct 31 19:53:40.166697 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.160089    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.200927 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.196448    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"]
Oct 31 19:53:40.241522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.241489    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9902ffd9-openshift-default-dockercfg-9t6ft\""
Oct 31 19:53:40.288978 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.288939    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcc8g\" (UniqueName: \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-kube-api-access-gcc8g\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.289142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.289017    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-podinfo\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.289142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.289087    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-credential-socket\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.289249 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.289160    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-certs\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.289290 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.289249    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-data\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.289344 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.289319    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istiod-ca-cert\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.289397 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.289382    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-token\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.289442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.289417    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-envoy\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.289442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.289437    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-socket\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.414823    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcc8g\" (UniqueName: \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-kube-api-access-gcc8g\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.414952    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-podinfo\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.415044    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-credential-socket\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.415140    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-certs\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.415258    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-data\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.415346    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istiod-ca-cert\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.415416    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-token\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.415492    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-envoy\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.415525    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-socket\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.416143    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-socket\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.418877    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:40.419675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.419500    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-podinfo\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.420458 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.419794    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-credential-socket\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.420458 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.420046    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-certs\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.420458 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.420265    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-data\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.427685 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.420873    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istiod-ca-cert\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.445771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.438128    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-envoy\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.477754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.477714    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcc8g\" (UniqueName: \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-kube-api-access-gcc8g\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.488055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.487998    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-token\") pod \"e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") " pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.491148 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.490944    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:40.742903 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.742804    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:40.969164 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.969118    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"]
Oct 31 19:53:40.993838 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:53:40.993774    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0617b1_6e0f_4f50_8634_00d4af4d0133.slice/crio-11e2a2f039a0b6410832c5285ebe956ed298d64563d7a0de76a93adce96ec49f WatchSource:0}: Error finding container 11e2a2f039a0b6410832c5285ebe956ed298d64563d7a0de76a93adce96ec49f: Status 404 returned error can't find the container with id 11e2a2f039a0b6410832c5285ebe956ed298d64563d7a0de76a93adce96ec49f
Oct 31 19:53:40.997286 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.997251    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:40.997372 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.997321    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:40.997372 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:40.997355    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:41.036905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:41.036862    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:41.438702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:41.438661    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" event={"ID":"2c0617b1-6e0f-4f50-8634-00d4af4d0133","Type":"ContainerStarted","Data":"9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138"}
Oct 31 19:53:41.438890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:41.438727    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" event={"ID":"2c0617b1-6e0f-4f50-8634-00d4af4d0133","Type":"ContainerStarted","Data":"11e2a2f039a0b6410832c5285ebe956ed298d64563d7a0de76a93adce96ec49f"}
Oct 31 19:53:41.491806 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:41.491772    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:41.559448 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:41.559389    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" podStartSLOduration=1.55937306 podStartE2EDuration="1.55937306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:41.555746094 +0000 UTC m=+2240.085280165" watchObservedRunningTime="2026-10-31 19:53:41.55937306 +0000 UTC m=+2240.088907113"
Oct 31 19:53:41.611544 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:41.611496    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1f2098-72a2-441b-a918-895637f761cf" path="/var/lib/kubelet/pods/2d1f2098-72a2-441b-a918-895637f761cf/volumes"
Oct 31 19:53:42.049378 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:42.049344    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"]
Oct 31 19:53:42.055777 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:42.055742    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23f86095-mnm6m"]
Oct 31 19:53:42.497285 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:42.497256    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:43.058896 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:43.058856    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"]
Oct 31 19:53:43.064095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:43.064064    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a6940a-58xns"]
Oct 31 19:53:43.455693 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:43.455626    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:43.456956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:43.456933    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:53:43.605367 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:43.605327    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6489a436-a0d9-43c5-a21d-414f083ee94f" path="/var/lib/kubelet/pods/6489a436-a0d9-43c5-a21d-414f083ee94f/volumes"
Oct 31 19:53:43.606132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:43.606106    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb48c2a-64c3-493e-bae0-fd287b943cbe" path="/var/lib/kubelet/pods/9eb48c2a-64c3-493e-bae0-fd287b943cbe/volumes"
Oct 31 19:53:44.077347 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:44.077292    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:53:44.250193 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:44.250154    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:45.516373 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:45.516325    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:45.760590 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:45.760541    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:47.298161 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.298127    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:47.535817 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.535780    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"]
Oct 31 19:53:47.540973 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.540947    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:53:47.544962 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.544914    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:47.603201 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.603165    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"]
Oct 31 19:53:47.647080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.647043    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-7c0d34-dockercfg-nkj57\""
Oct 31 19:53:47.647298 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.647161    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-7c0d34\""
Oct 31 19:53:47.691675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.672789    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7659e-2773-48f1-a0e3-dda638c0c23a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh\" (UID: \"3ba7659e-2773-48f1-a0e3-dda638c0c23a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:53:47.691675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.672871    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94kk9\" (UniqueName: \"kubernetes.io/projected/3ba7659e-2773-48f1-a0e3-dda638c0c23a-kube-api-access-94kk9\") pod \"payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh\" (UID: \"3ba7659e-2773-48f1-a0e3-dda638c0c23a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:53:47.784550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.768699    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"]
Oct 31 19:53:47.784550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.774980    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7659e-2773-48f1-a0e3-dda638c0c23a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh\" (UID: \"3ba7659e-2773-48f1-a0e3-dda638c0c23a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:53:47.784550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.775040    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94kk9\" (UniqueName: \"kubernetes.io/projected/3ba7659e-2773-48f1-a0e3-dda638c0c23a-kube-api-access-94kk9\") pod \"payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh\" (UID: \"3ba7659e-2773-48f1-a0e3-dda638c0c23a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:53:47.784550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.776571    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7659e-2773-48f1-a0e3-dda638c0c23a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh\" (UID: \"3ba7659e-2773-48f1-a0e3-dda638c0c23a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:53:47.784550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.783007    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"]
Oct 31 19:53:47.784550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.783154    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:53:47.852903 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.852868    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94kk9\" (UniqueName: \"kubernetes.io/projected/3ba7659e-2773-48f1-a0e3-dda638c0c23a-kube-api-access-94kk9\") pod \"payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh\" (UID: \"3ba7659e-2773-48f1-a0e3-dda638c0c23a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:53:47.860422 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.860389    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:53:47.877337 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.877301    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vr5\" (UniqueName: \"kubernetes.io/projected/d53c7a5b-0587-4a65-bc21-4f637e6ec278-kube-api-access-d2vr5\") pod \"payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv\" (UID: \"d53c7a5b-0587-4a65-bc21-4f637e6ec278\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:53:47.877890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.877461    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53c7a5b-0587-4a65-bc21-4f637e6ec278-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv\" (UID: \"d53c7a5b-0587-4a65-bc21-4f637e6ec278\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:53:47.979490 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.979447    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vr5\" (UniqueName: \"kubernetes.io/projected/d53c7a5b-0587-4a65-bc21-4f637e6ec278-kube-api-access-d2vr5\") pod \"payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv\" (UID: \"d53c7a5b-0587-4a65-bc21-4f637e6ec278\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:53:47.979701 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.979567    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53c7a5b-0587-4a65-bc21-4f637e6ec278-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv\" (UID: \"d53c7a5b-0587-4a65-bc21-4f637e6ec278\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:53:47.980568 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:47.980533    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53c7a5b-0587-4a65-bc21-4f637e6ec278-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv\" (UID: \"d53c7a5b-0587-4a65-bc21-4f637e6ec278\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:53:48.039670 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:48.039624    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vr5\" (UniqueName: \"kubernetes.io/projected/d53c7a5b-0587-4a65-bc21-4f637e6ec278-kube-api-access-d2vr5\") pod \"payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv\" (UID: \"d53c7a5b-0587-4a65-bc21-4f637e6ec278\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:53:48.118889 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:48.118853    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:53:48.265235 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:48.264773    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"]
Oct 31 19:53:48.266549 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:53:48.266516    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba7659e_2773_48f1_a0e3_dda638c0c23a.slice/crio-e33e1ddfc7f6f988748b40bdb0a2387095a8e6acccd262898369e6a6688c5afe WatchSource:0}: Error finding container e33e1ddfc7f6f988748b40bdb0a2387095a8e6acccd262898369e6a6688c5afe: Status 404 returned error can't find the container with id e33e1ddfc7f6f988748b40bdb0a2387095a8e6acccd262898369e6a6688c5afe
Oct 31 19:53:48.496951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:48.496908    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh" event={"ID":"3ba7659e-2773-48f1-a0e3-dda638c0c23a","Type":"ContainerStarted","Data":"16ad8abc7b4745e91b5e6f1a8dfc71a4cc9fab3fd4cb1695ab253384d4ddfbc7"}
Oct 31 19:53:48.497350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:48.496972    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:53:48.497350 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:48.496996    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh" event={"ID":"3ba7659e-2773-48f1-a0e3-dda638c0c23a","Type":"ContainerStarted","Data":"e33e1ddfc7f6f988748b40bdb0a2387095a8e6acccd262898369e6a6688c5afe"}
Oct 31 19:53:48.514029 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:53:48.513995    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53c7a5b_0587_4a65_bc21_4f637e6ec278.slice/crio-a41f9df595f872f518a278a89a24ec590161297bd88120f95860c16c165e32a2 WatchSource:0}: Error finding container a41f9df595f872f518a278a89a24ec590161297bd88120f95860c16c165e32a2: Status 404 returned error can't find the container with id a41f9df595f872f518a278a89a24ec590161297bd88120f95860c16c165e32a2
Oct 31 19:53:48.522700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:48.520011    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"]
Oct 31 19:53:48.614841 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:48.614791    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh" podStartSLOduration=1.614774194 podStartE2EDuration="1.614774194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:48.612603761 +0000 UTC m=+2247.142137832" watchObservedRunningTime="2026-10-31 19:53:48.614774194 +0000 UTC m=+2247.144308273"
Oct 31 19:53:49.506550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:49.506509    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv" event={"ID":"d53c7a5b-0587-4a65-bc21-4f637e6ec278","Type":"ContainerStarted","Data":"ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180"}
Oct 31 19:53:49.507058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:49.506572    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv" event={"ID":"d53c7a5b-0587-4a65-bc21-4f637e6ec278","Type":"ContainerStarted","Data":"a41f9df595f872f518a278a89a24ec590161297bd88120f95860c16c165e32a2"}
Oct 31 19:53:49.507058 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:49.506943    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:53:49.622332 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:49.622257    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv" podStartSLOduration=2.62223743 podStartE2EDuration="2.62223743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:49.621338563 +0000 UTC m=+2248.150872617" watchObservedRunningTime="2026-10-31 19:53:49.62223743 +0000 UTC m=+2248.151771483"
Oct 31 19:53:49.709967 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:49.709930    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:49.926673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:49.925882    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:53.487690 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:53.487632    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:55.120025 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:55.119145    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:55.331024 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:55.330990    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:59.510754 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:59.510719    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:53:59.684637 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:53:59.684603    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:00.390167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:00.390123    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:00.518668 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:00.518625    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:54:01.051139 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:01.051102    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"]
Oct 31 19:54:01.056542 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:01.056511    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12d17959-c6g2r"]
Oct 31 19:54:01.600609 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:01.600576    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89545a2-6e00-4f0e-bc41-5e6733ad9b93" path="/var/lib/kubelet/pods/c89545a2-6e00-4f0e-bc41-5e6733ad9b93/volumes"
Oct 31 19:54:05.302195 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:05.302154    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:05.396890 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:54:05.396851    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:54:09.632460 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:09.632412    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:11.777700 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:11.776495    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"]
Oct 31 19:54:11.807731 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:11.807687    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-abb7688a-c8wjr"]
Oct 31 19:54:12.266908 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:12.266872    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:12.485920 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:12.485889    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:12.704199 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:12.704151    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:13.064985 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:13.064890    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"]
Oct 31 19:54:13.112406 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:13.112367    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4f5dc5-c6l7f"]
Oct 31 19:54:13.601520 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:13.601481    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4000f9e9-71e6-4053-8077-e4c19f856946" path="/var/lib/kubelet/pods/4000f9e9-71e6-4053-8077-e4c19f856946/volumes"
Oct 31 19:54:13.602162 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:13.602139    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1cb72c2-4899-4301-b457-c594a5f52437" path="/var/lib/kubelet/pods/d1cb72c2-4899-4301-b457-c594a5f52437/volumes"
Oct 31 19:54:13.639290 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:13.639256    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:13.871257 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:13.871175    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:16.252055 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:16.252015    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:18.348542 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:18.348487    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:18.799339 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:18.799241    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:19.090461 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:19.090390    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:19.247270 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:19.247224    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"]
Oct 31 19:54:19.297781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:19.297745    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84fd8e82-tm4r8"]
Oct 31 19:54:19.600712 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:19.600636    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2065539-2931-45f4-8110-dcb75776f711" path="/var/lib/kubelet/pods/f2065539-2931-45f4-8110-dcb75776f711/volumes"
Oct 31 19:54:19.804287 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:19.802201    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:20.039771 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:20.036182    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:20.255422 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:20.255386    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:54:20.733285 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:20.733230    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:21.485505 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:21.485457    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:21.937197 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:21.937162    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:22.837132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.837102    5006 scope.go:122] "RemoveContainer" containerID="4ef9e962132b8a5ee4e776c2e5cf328fb21081a3d159240ff009e302897b7b61"
Oct 31 19:54:22.848625 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.848603    5006 scope.go:122] "RemoveContainer" containerID="681c4ffc47ee251b834ed5299b2c4c77551d66f7d40a567a6c530a3dd532c43e"
Oct 31 19:54:22.862729 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.862699    5006 scope.go:122] "RemoveContainer" containerID="8fb7b90af1cbe8ecdbfb6f923dbf4e5f33bbffd3c2a17f1592f797c8c937012c"
Oct 31 19:54:22.879967 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.878974    5006 scope.go:122] "RemoveContainer" containerID="6a78df03d79b043208e8c5eb44bd282282091d403efe81dd1331ff364131ce25"
Oct 31 19:54:22.897104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.897078    5006 scope.go:122] "RemoveContainer" containerID="494e260579a722d9ab0a85ca0553c0ea3fae3926459b523eec8eabb041e3e94e"
Oct 31 19:54:22.912073 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.912020    5006 scope.go:122] "RemoveContainer" containerID="be58b242282c3c424be09ba9bbef21094ff1267a8ea4f78ae4305e7d5f40eb00"
Oct 31 19:54:22.928610 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.928559    5006 scope.go:122] "RemoveContainer" containerID="3f7e8ac27f27a8528adb4b11b0a306ba1029fe97ad5c3b14af4f923948ed5285"
Oct 31 19:54:22.948593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.944953    5006 scope.go:122] "RemoveContainer" containerID="1f938571867107413e6b557a63e3b6bd03123aa4ee608523377f826c639fd3eb"
Oct 31 19:54:22.974583 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.974554    5006 scope.go:122] "RemoveContainer" containerID="a2ea4fba815c9a592d78e7bb5e687d36dda651761329aa04633994614acb242c"
Oct 31 19:54:22.986245 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.986224    5006 scope.go:122] "RemoveContainer" containerID="c3a4309a0552002ee784dcf44896cb318b00662105a2084bf490fad2896ffedb"
Oct 31 19:54:22.999169 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:22.999146    5006 scope.go:122] "RemoveContainer" containerID="90bf54e3979b8cbf6d3e935b4c339201dbabbe00dd564a9daff06dfd1415864c"
Oct 31 19:54:23.011387 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:23.011363    5006 scope.go:122] "RemoveContainer" containerID="b98b5f5f506a94f6dcc8cec32889afac06a2f30043c4ea09539de66f48783b7f"
Oct 31 19:54:23.021979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:23.021955    5006 scope.go:122] "RemoveContainer" containerID="307bf2e37be7013104ca25042c7087cb9964261ab0a1e494d0aa7bc41ff83a27"
Oct 31 19:54:23.078343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:23.078317    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:23.421412 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:23.418100    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:23.952734 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:23.952697    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:24.049617 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:24.049575    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"]
Oct 31 19:54:24.098241 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:24.098200    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87cc9a2-nt5l5"]
Oct 31 19:54:24.474915 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:24.474877    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:25.600287 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:25.600249    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59d0fd17-18c7-4c99-9731-120dfee085d7" path="/var/lib/kubelet/pods/59d0fd17-18c7-4c99-9731-120dfee085d7/volumes"
Oct 31 19:54:25.874542 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:25.874450    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:26.107017 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:26.106982    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:27.920513 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:27.920478    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:28.771539 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:28.771500    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:29.009310 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:29.009273    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:29.328916 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:29.328857    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:29.550965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:29.550930    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:30.312336 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:30.312285    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:30.531324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:30.531283    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:34.867642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:34.867557    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:35.395375 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:54:35.395332    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:54:35.839946 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:35.839913    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:39.791442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:39.791405    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:40.033639 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:40.033598    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:40.273557 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:40.273517    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:44.829636 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:44.829592    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:45.074442 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:45.074406    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:50.056085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:50.056036    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:55.168014 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:54:55.167933    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:00.261694 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:00.261643    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:04.191470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:04.191423    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:55:05.398486 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:55:05.398443    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:55:05.470958 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:05.470919    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:06.180049 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:06.180016    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:06.643835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:06.643802    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:07.874025 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:07.873969    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:08.363299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:08.363269    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:11.227284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:11.227241    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:12.374267 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:12.374222    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:12.611784 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:12.611744    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:12.843485 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:12.843447    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:15.476284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:15.476192    5006 generic.go:356] "Generic (PLEG): container finished" podID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerID="69db270188f1bec2cd5b94539fa5301e1d35483d0ba12cb38451f232fe23eaae" exitCode=0
Oct 31 19:55:15.476284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:15.476240    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" event={"ID":"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78","Type":"ContainerDied","Data":"69db270188f1bec2cd5b94539fa5301e1d35483d0ba12cb38451f232fe23eaae"}
Oct 31 19:55:15.476793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:15.476779    5006 scope.go:122] "RemoveContainer" containerID="69db270188f1bec2cd5b94539fa5301e1d35483d0ba12cb38451f232fe23eaae"
Oct 31 19:55:15.478589 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:15.478556    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:15.478706 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:15.478632    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:15.478706 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:15.478699    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:15.755632 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:15.755543    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:15.985287 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:15.985250    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:16.499177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:16.499129    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" event={"ID":"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78","Type":"ContainerStarted","Data":"72d483d274e5f24c3ecdc7854a588eee66e68465599afa88c6a4beb69658699a"}
Oct 31 19:55:16.864943 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:16.864905    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:17.263061 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:17.262967    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:55:17.263609 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:17.263255    5006 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:55:17.263609 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:17.263305    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 19:55:18.263524 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:18.263485    5006 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:55:18.264002 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:18.263539    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 19:55:19.263533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:19.263496    5006 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:55:19.263993 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:19.263572    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 19:55:20.263262 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:20.263217    5006 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:55:20.263482 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:20.263296    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 19:55:21.262908 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:21.262860    5006 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:55:21.263334 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:21.262941    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 19:55:21.626189 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:21.626144    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:21.857671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:21.854293    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:22.076487 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:22.076397    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:22.267566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:22.267532    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:55:22.268057 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:22.268036    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:55:22.268768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:22.268751    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 19:55:23.606768 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:23.606735    5006 scope.go:122] "RemoveContainer" containerID="4f3667bfacfdcd43b0314837aac1b6af25968f82ca84c52b62503ffef9b1edbc"
Oct 31 19:55:23.617599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:23.617565    5006 scope.go:122] "RemoveContainer" containerID="504d14cfff821e7786fcee6e2ca53da99b7157052f9afbc9e96aafb2fdd9e74f"
Oct 31 19:55:23.628282 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:23.628257    5006 scope.go:122] "RemoveContainer" containerID="ff372ccd1aaf1272c7b120cdf22ff78d1226aad90f9971fa142eb39118298fd0"
Oct 31 19:55:23.640356 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:23.640325    5006 scope.go:122] "RemoveContainer" containerID="75199fde0794622fe504ac5d313aabc01f6097f5a5efa4b08584c95ba0f81d7b"
Oct 31 19:55:23.651279 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:23.651256    5006 scope.go:122] "RemoveContainer" containerID="05e822b841034854489dea9351e05267e73169da4e9fa5f1a8a151bb41c2e2f4"
Oct 31 19:55:23.661534 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:23.661512    5006 scope.go:122] "RemoveContainer" containerID="53e738e930b99fd4bd78f6286a28e6081f2f0a86b660c2f796d6a1ce1aecb8ca"
Oct 31 19:55:23.672285 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:23.672263    5006 scope.go:122] "RemoveContainer" containerID="e946974e60a8acff4e67a7d6f220cc148a5be4a2cb3f2e3cf6539dba43bbf58d"
Oct 31 19:55:23.682287 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:23.682264    5006 scope.go:122] "RemoveContainer" containerID="d5449d16922fa5decf8fefcb26f44fe324a3f09203691ee20b4e2e745afec5e1"
Oct 31 19:55:23.693047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:23.693023    5006 scope.go:122] "RemoveContainer" containerID="3223338bda25244d81412e8cada002fcd956d0ffbe29c77f4cab996208809614"
Oct 31 19:55:24.140919 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:24.140886    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:24.847674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:24.844504    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:26.136620 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:26.136584    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:26.364908 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:26.364872    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:26.803453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:26.803253    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:30.670328 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:30.670289    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:32.083694 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:32.083641    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:33.955343 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:33.955305    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:34.216785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:34.216685    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:35.398077 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:55:35.398038    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:55:37.203618 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:37.203576    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:37.427420 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:37.427389    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:41.993105 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:41.993032    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:43.048393 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:43.048355    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"]
Oct 31 19:55:43.098183 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:43.098140    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa988fb4-jc8pj"]
Oct 31 19:55:43.600870 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:43.600832    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a37408c-91cc-4cf4-b17c-cea5908dfd47" path="/var/lib/kubelet/pods/3a37408c-91cc-4cf4-b17c-cea5908dfd47/volumes"
Oct 31 19:55:46.869956 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:46.869923    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:51.775250 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:51.775212    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:56.825274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:56.825227    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:58.399919 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:58.399868    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:58.865675 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:58.864738    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:58.942742 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:58.938028    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:55:59.738695 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:55:59.721508    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:00.210219 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.210179    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:00.228674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.224724    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"]
Oct 31 19:56:00.254671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.250946    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:00.304715 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.304640    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:00.305428 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.305398    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5jn\" (UniqueName: \"kubernetes.io/projected/057c2dc1-46d2-49f5-b193-81139d65d2d0-kube-api-access-6c5jn\") pod \"maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:00.305819 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.305795    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:00.318558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.315096    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:56:00.419672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.375527    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"]
Oct 31 19:56:00.419992 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.385824    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:56:00.420137 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.385959    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:56:00.425869 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.390715    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:56:00.433783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.407830    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:00.433783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.418264    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:00.433783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.431857    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5jn\" (UniqueName: \"kubernetes.io/projected/057c2dc1-46d2-49f5-b193-81139d65d2d0-kube-api-access-6c5jn\") pod \"maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:00.433783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.432079    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:00.442421 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.442389    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:00.491098 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.491012    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5jn\" (UniqueName: \"kubernetes.io/projected/057c2dc1-46d2-49f5-b193-81139d65d2d0-kube-api-access-6c5jn\") pod \"maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:00.593604 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:00.593562    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:01.024153 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:56:01.024102    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057c2dc1_46d2_49f5_b193_81139d65d2d0.slice/crio-3cfa966027e9149cb008264fa72cedef00f20b77d9fb4469b09e45eeecd02f64 WatchSource:0}: Error finding container 3cfa966027e9149cb008264fa72cedef00f20b77d9fb4469b09e45eeecd02f64: Status 404 returned error can't find the container with id 3cfa966027e9149cb008264fa72cedef00f20b77d9fb4469b09e45eeecd02f64
Oct 31 19:56:01.027284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:01.027248    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"]
Oct 31 19:56:02.038001 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:02.037962    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd" event={"ID":"057c2dc1-46d2-49f5-b193-81139d65d2d0","Type":"ContainerStarted","Data":"b3f6a87f33732df8b9c14345a287eeab828637984a626b8525c9e0efa2981b75"}
Oct 31 19:56:02.038535 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:02.038027    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd" event={"ID":"057c2dc1-46d2-49f5-b193-81139d65d2d0","Type":"ContainerStarted","Data":"3cfa966027e9149cb008264fa72cedef00f20b77d9fb4469b09e45eeecd02f64"}
Oct 31 19:56:02.153197 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:02.153134    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd" podStartSLOduration=2.153119496 podStartE2EDuration="2.153119496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:02.152837474 +0000 UTC m=+2380.682371528" watchObservedRunningTime="2026-10-31 19:56:02.153119496 +0000 UTC m=+2380.682653540"
Oct 31 19:56:02.675323 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:02.673697    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:03.047175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:03.047091    5006 generic.go:356] "Generic (PLEG): container finished" podID="057c2dc1-46d2-49f5-b193-81139d65d2d0" containerID="b3f6a87f33732df8b9c14345a287eeab828637984a626b8525c9e0efa2981b75" exitCode=0
Oct 31 19:56:03.047175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:03.047141    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd" event={"ID":"057c2dc1-46d2-49f5-b193-81139d65d2d0","Type":"ContainerDied","Data":"b3f6a87f33732df8b9c14345a287eeab828637984a626b8525c9e0efa2981b75"}
Oct 31 19:56:04.247950 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.247915    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:04.269033 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.269007    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:04.431000 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.430964    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057c2dc1-46d2-49f5-b193-81139d65d2d0-kube-api-access-6c5jn\" (UniqueName: \"kubernetes.io/projected/057c2dc1-46d2-49f5-b193-81139d65d2d0-kube-api-access-6c5jn\") pod \"057c2dc1-46d2-49f5-b193-81139d65d2d0\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") "
Oct 31 19:56:04.431216 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.431030    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-openshift-ca\") pod \"057c2dc1-46d2-49f5-b193-81139d65d2d0\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") "
Oct 31 19:56:04.431216 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.431061    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-xks-ca\") pod \"057c2dc1-46d2-49f5-b193-81139d65d2d0\" (UID: \"057c2dc1-46d2-49f5-b193-81139d65d2d0\") "
Oct 31 19:56:04.431517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.431484    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-openshift-ca" pod "057c2dc1-46d2-49f5-b193-81139d65d2d0" (UID: "057c2dc1-46d2-49f5-b193-81139d65d2d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:04.431911 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.431883    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:04.433405 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.433378    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-xks-ca" pod "057c2dc1-46d2-49f5-b193-81139d65d2d0" (UID: "057c2dc1-46d2-49f5-b193-81139d65d2d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:56:04.433767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.433746    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057c2dc1-46d2-49f5-b193-81139d65d2d0-kube-api-access-6c5jn" pod "057c2dc1-46d2-49f5-b193-81139d65d2d0" (UID: "057c2dc1-46d2-49f5-b193-81139d65d2d0"). InnerVolumeSpecName "kube-api-access-6c5jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:04.475789 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.475755    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:04.533968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.533929    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c5jn\" (UniqueName: \"kubernetes.io/projected/057c2dc1-46d2-49f5-b193-81139d65d2d0-kube-api-access-6c5jn\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:04.533968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:04.533964    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057c2dc1-46d2-49f5-b193-81139d65d2d0-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:05.063966 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:05.063934    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"
Oct 31 19:56:05.064159 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:05.063968    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd" event={"ID":"057c2dc1-46d2-49f5-b193-81139d65d2d0","Type":"ContainerDied","Data":"3cfa966027e9149cb008264fa72cedef00f20b77d9fb4469b09e45eeecd02f64"}
Oct 31 19:56:05.064159 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:05.064013    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cfa966027e9149cb008264fa72cedef00f20b77d9fb4469b09e45eeecd02f64"
Oct 31 19:56:05.395604 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:56:05.395565    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:56:07.045505 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:07.045458    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:08.220772 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:08.220731    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:08.449671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:08.448312    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:10.385464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:10.385421    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:12.141281 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:12.141238    5006 generic.go:356] "Generic (PLEG): container finished" podID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerID="9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138" exitCode=0
Oct 31 19:56:12.141947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:12.141387    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" event={"ID":"2c0617b1-6e0f-4f50-8634-00d4af4d0133","Type":"ContainerDied","Data":"9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138"}
Oct 31 19:56:12.142598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:12.142527    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:12.142867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:12.142843    5006 scope.go:122] "RemoveContainer" containerID="9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138"
Oct 31 19:56:12.145482 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:12.145446    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:56:12.145593 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:12.145531    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:56:12.145668 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:12.145590    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:56:12.395396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:12.395306    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:13.198677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:13.197724    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" event={"ID":"2c0617b1-6e0f-4f50-8634-00d4af4d0133","Type":"ContainerStarted","Data":"a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313"}
Oct 31 19:56:13.477732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:13.476776    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:56:13.502515 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:13.502470    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:56:13.515859 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:13.515820    5006 patch_prober.go:29] interesting pod/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:56:13.516166 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:13.516131    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 19:56:13.725673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:13.709013    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:14.492349 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:14.492300    5006 patch_prober.go:29] interesting pod/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:56:14.492931 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:14.492389    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 19:56:15.491879 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:15.491834    5006 patch_prober.go:29] interesting pod/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:56:15.492088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:15.491923    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 19:56:16.492234 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:16.492197    5006 patch_prober.go:29] interesting pod/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:56:16.492719 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:16.492262    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 19:56:17.492324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:17.492287    5006 patch_prober.go:29] interesting pod/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:56:17.492822 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:17.492353    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 19:56:18.492391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:18.492354    5006 patch_prober.go:29] interesting pod/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 19:56:18.492867 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:18.492420    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 19:56:18.733923 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:18.733883    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:19.497274 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:19.497239    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:56:19.498368 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:19.498347    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 19:56:22.091967 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:22.091935    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:56:22.170691 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:22.170662    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 19:56:23.399806 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:23.399770    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:23.624060 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:23.624023    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:23.831965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:23.831864    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:25.183890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:25.183859    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:28.516173 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:28.516134    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:33.370764 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:33.370688    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:35.287265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:35.287217    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:56:35.398193 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:56:35.398154    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:56:38.299865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:38.299815    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:42.613674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:42.610499    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:44.439805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:44.439769    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:44.653613 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:44.653578    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:45.383140 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:45.383098    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:45.704775 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:45.704633    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:45.865979 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:45.865946    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"]
Oct 31 19:56:45.915738 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:45.915699    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f37f6f32-r6vtd"]
Oct 31 19:56:46.214595 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:46.214566    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:46.493723 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:46.493613    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:46.726241 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:46.726207    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:47.516237 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:47.516192    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:47.757719 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:47.751563    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057c2dc1-46d2-49f5-b193-81139d65d2d0" path="/var/lib/kubelet/pods/057c2dc1-46d2-49f5-b193-81139d65d2d0/volumes"
Oct 31 19:56:47.809461 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:47.809367    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:50.109537 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:50.109498    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:51.731758 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:51.731722    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:52.041786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:52.041691    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:52.733533 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:52.733502    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"]
Oct 31 19:56:52.734095 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:52.733741    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy" containerID="cri-o://39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5" gracePeriod=30
Oct 31 19:56:52.864813 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:52.864778    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:53.042411 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:53.042321    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"]
Oct 31 19:56:53.042591 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:53.042561    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy" containerID="cri-o://2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7" gracePeriod=30
Oct 31 19:56:53.260353 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:53.260283    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:53.489774 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:53.489733    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:54.459425 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:54.459391    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:54.602258 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:54.602219    5006 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:56:54.602444 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:54.602285    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:56:55.069942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:55.069903    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:55.331730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:55.331259    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:55.711671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:55.711338    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:56.228570 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:56.228530    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:56.449080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:56.449043    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:57.940598 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:57.940551    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:58.036538 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.036501    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:56:58.059765 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.059665    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-data\") pod \"007305fa-ae8e-4596-9dde-778b8fad4957\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") "
Oct 31 19:56:58.059858 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.059761    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-envoy\") pod \"007305fa-ae8e-4596-9dde-778b8fad4957\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") "
Oct 31 19:56:58.060003 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.059954    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-data" pod "007305fa-ae8e-4596-9dde-778b8fad4957" (UID: "007305fa-ae8e-4596-9dde-778b8fad4957"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.060080 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.060060    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-certs\") pod \"007305fa-ae8e-4596-9dde-778b8fad4957\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") "
Oct 31 19:56:58.060151 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.060114    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-kube-api-access-dbjw8\" (UniqueName: \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-kube-api-access-dbjw8\") pod \"007305fa-ae8e-4596-9dde-778b8fad4957\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") "
Oct 31 19:56:58.060151 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.060143    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/007305fa-ae8e-4596-9dde-778b8fad4957-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007305fa-ae8e-4596-9dde-778b8fad4957-istio-podinfo\") pod \"007305fa-ae8e-4596-9dde-778b8fad4957\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") "
Oct 31 19:56:58.060297 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.060276    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-socket\") pod \"007305fa-ae8e-4596-9dde-778b8fad4957\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") "
Oct 31 19:56:58.061023 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.060362    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007305fa-ae8e-4596-9dde-778b8fad4957-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007305fa-ae8e-4596-9dde-778b8fad4957-istiod-ca-cert\") pod \"007305fa-ae8e-4596-9dde-778b8fad4957\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") "
Oct 31 19:56:58.061023 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.060428    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-istio-token\" (UniqueName: \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-istio-token\") pod \"007305fa-ae8e-4596-9dde-778b8fad4957\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") "
Oct 31 19:56:58.061023 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.060461    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-credential-socket\") pod \"007305fa-ae8e-4596-9dde-778b8fad4957\" (UID: \"007305fa-ae8e-4596-9dde-778b8fad4957\") "
Oct 31 19:56:58.061023 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.060982    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-certs" pod "007305fa-ae8e-4596-9dde-778b8fad4957" (UID: "007305fa-ae8e-4596-9dde-778b8fad4957"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.061338 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.061049    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-credential-socket" pod "007305fa-ae8e-4596-9dde-778b8fad4957" (UID: "007305fa-ae8e-4596-9dde-778b8fad4957"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.061338 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.061185    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-socket" pod "007305fa-ae8e-4596-9dde-778b8fad4957" (UID: "007305fa-ae8e-4596-9dde-778b8fad4957"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.061338 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.061258    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007305fa-ae8e-4596-9dde-778b8fad4957-istiod-ca-cert" pod "007305fa-ae8e-4596-9dde-778b8fad4957" (UID: "007305fa-ae8e-4596-9dde-778b8fad4957"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:58.061466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.061368    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.061466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.061383    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.061466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.061393    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.061466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.061402    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.063157 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.063118    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-envoy" pod "007305fa-ae8e-4596-9dde-778b8fad4957" (UID: "007305fa-ae8e-4596-9dde-778b8fad4957"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.063704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.063677    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-istio-token" pod "007305fa-ae8e-4596-9dde-778b8fad4957" (UID: "007305fa-ae8e-4596-9dde-778b8fad4957"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:58.064131 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.064107    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/007305fa-ae8e-4596-9dde-778b8fad4957-istio-podinfo" pod "007305fa-ae8e-4596-9dde-778b8fad4957" (UID: "007305fa-ae8e-4596-9dde-778b8fad4957"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:58.064891 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.064866    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-kube-api-access-dbjw8" pod "007305fa-ae8e-4596-9dde-778b8fad4957" (UID: "007305fa-ae8e-4596-9dde-778b8fad4957"). InnerVolumeSpecName "kube-api-access-dbjw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:58.155029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.154989    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:58.165829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.165697    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007305fa-ae8e-4596-9dde-778b8fad4957-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.165829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.165725    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbjw8\" (UniqueName: \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-kube-api-access-dbjw8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.165829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.165741    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007305fa-ae8e-4596-9dde-778b8fad4957-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.165829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.165754    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007305fa-ae8e-4596-9dde-778b8fad4957-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.165829 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.165765    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007305fa-ae8e-4596-9dde-778b8fad4957-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.193767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.193726    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 19:56:58.193968 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.193809    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 19:56:58.374202 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.374173    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:56:58.469259 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469218    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-credential-socket\") pod \"aba6c1e9-986e-44e5-a460-030a11eacb91\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") "
Oct 31 19:56:58.469454 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469293    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-data\") pod \"aba6c1e9-986e-44e5-a460-030a11eacb91\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") "
Oct 31 19:56:58.469454 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469322    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-kube-api-access-vj68f\" (UniqueName: \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-kube-api-access-vj68f\") pod \"aba6c1e9-986e-44e5-a460-030a11eacb91\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") "
Oct 31 19:56:58.469454 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469429    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-podinfo\") pod \"aba6c1e9-986e-44e5-a460-030a11eacb91\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") "
Oct 31 19:56:58.469626 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469470    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-envoy\") pod \"aba6c1e9-986e-44e5-a460-030a11eacb91\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") "
Oct 31 19:56:58.469626 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469572    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-token\" (UniqueName: \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-token\") pod \"aba6c1e9-986e-44e5-a460-030a11eacb91\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") "
Oct 31 19:56:58.469833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469627    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-data" pod "aba6c1e9-986e-44e5-a460-030a11eacb91" (UID: "aba6c1e9-986e-44e5-a460-030a11eacb91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.469833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469713    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-certs\") pod \"aba6c1e9-986e-44e5-a460-030a11eacb91\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") "
Oct 31 19:56:58.469833 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469794    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-socket\") pod \"aba6c1e9-986e-44e5-a460-030a11eacb91\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") "
Oct 31 19:56:58.469991 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469870    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba6c1e9-986e-44e5-a460-030a11eacb91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba6c1e9-986e-44e5-a460-030a11eacb91-istiod-ca-cert\") pod \"aba6c1e9-986e-44e5-a460-030a11eacb91\" (UID: \"aba6c1e9-986e-44e5-a460-030a11eacb91\") "
Oct 31 19:56:58.470040 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.469990    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-credential-socket" pod "aba6c1e9-986e-44e5-a460-030a11eacb91" (UID: "aba6c1e9-986e-44e5-a460-030a11eacb91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.470040 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.470026    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-certs" pod "aba6c1e9-986e-44e5-a460-030a11eacb91" (UID: "aba6c1e9-986e-44e5-a460-030a11eacb91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.470229 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.470206    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-socket" pod "aba6c1e9-986e-44e5-a460-030a11eacb91" (UID: "aba6c1e9-986e-44e5-a460-030a11eacb91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.470454 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.470429    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba6c1e9-986e-44e5-a460-030a11eacb91-istiod-ca-cert" pod "aba6c1e9-986e-44e5-a460-030a11eacb91" (UID: "aba6c1e9-986e-44e5-a460-030a11eacb91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:58.470765 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.470740    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.470851 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.470771    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.470851 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.470788    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba6c1e9-986e-44e5-a460-030a11eacb91-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.470851 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.470803    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.470851 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.470818    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.472067 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.472042    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-podinfo" pod "aba6c1e9-986e-44e5-a460-030a11eacb91" (UID: "aba6c1e9-986e-44e5-a460-030a11eacb91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:58.472172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.472077    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-kube-api-access-vj68f" pod "aba6c1e9-986e-44e5-a460-030a11eacb91" (UID: "aba6c1e9-986e-44e5-a460-030a11eacb91"). InnerVolumeSpecName "kube-api-access-vj68f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:58.472172 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.472107    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-token" pod "aba6c1e9-986e-44e5-a460-030a11eacb91" (UID: "aba6c1e9-986e-44e5-a460-030a11eacb91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:58.472402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.472377    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-envoy" pod "aba6c1e9-986e-44e5-a460-030a11eacb91" (UID: "aba6c1e9-986e-44e5-a460-030a11eacb91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.572300 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.572206    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj68f\" (UniqueName: \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-kube-api-access-vj68f\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.572300 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.572242    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.572300 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.572252    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.572300 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.572261    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba6c1e9-986e-44e5-a460-030a11eacb91-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.800568 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.800530    5006 generic.go:356] "Generic (PLEG): container finished" podID="007305fa-ae8e-4596-9dde-778b8fad4957" containerID="39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5" exitCode=0
Oct 31 19:56:58.800785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.800588    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"
Oct 31 19:56:58.800785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.800621    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" event={"ID":"007305fa-ae8e-4596-9dde-778b8fad4957","Type":"ContainerDied","Data":"39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5"}
Oct 31 19:56:58.800785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.800710    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59" event={"ID":"007305fa-ae8e-4596-9dde-778b8fad4957","Type":"ContainerDied","Data":"41a38127ba6411b39480ca1115cf6f593aacd6a2363577c20129f8cdc2dc100b"}
Oct 31 19:56:58.800785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.800740    5006 scope.go:122] "RemoveContainer" containerID="39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5"
Oct 31 19:56:58.802351 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.802323    5006 generic.go:356] "Generic (PLEG): container finished" podID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerID="2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7" exitCode=0
Oct 31 19:56:58.802504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.802389    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"
Oct 31 19:56:58.802504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.802435    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" event={"ID":"aba6c1e9-986e-44e5-a460-030a11eacb91","Type":"ContainerDied","Data":"2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7"}
Oct 31 19:56:58.802504 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.802480    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss" event={"ID":"aba6c1e9-986e-44e5-a460-030a11eacb91","Type":"ContainerDied","Data":"b79b4e60df0e6c8c7f2dd056a5ca6c2a95c781dd70e11d13477876e5214ece08"}
Oct 31 19:56:58.812918 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.812753    5006 scope.go:122] "RemoveContainer" containerID="39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5"
Oct 31 19:56:58.813176 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:56:58.813151    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5\": container with ID starting with 39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5 not found: ID does not exist" containerID="39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5"
Oct 31 19:56:58.813283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.813188    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5"} err="failed to get container status \"39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5\": rpc error: code = NotFound desc = could not find container \"39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5\": container with ID starting with 39c34bdbd225dcfbca330363faf50424c5b9531a06f6189feb18aab15d673ce5 not found: ID does not exist"
Oct 31 19:56:58.813283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.813216    5006 scope.go:122] "RemoveContainer" containerID="2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7"
Oct 31 19:56:58.824242 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.824214    5006 scope.go:122] "RemoveContainer" containerID="2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7"
Oct 31 19:56:58.824606 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:56:58.824586    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7\": container with ID starting with 2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7 not found: ID does not exist" containerID="2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7"
Oct 31 19:56:58.824687 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.824617    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7"} err="failed to get container status \"2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7\": rpc error: code = NotFound desc = could not find container \"2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7\": container with ID starting with 2c25396528c401b5e0fac1dc29ac5c7d765d27b8a9ce278f2d74d19a835decf7 not found: ID does not exist"
Oct 31 19:56:58.968781 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.968740    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"]
Oct 31 19:56:58.974823 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:58.974791    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-b8zss"]
Oct 31 19:56:59.084732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:59.084632    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"]
Oct 31 19:56:59.089054 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:59.089018    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-wfj59"]
Oct 31 19:56:59.600556 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:59.600522    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" path="/var/lib/kubelet/pods/007305fa-ae8e-4596-9dde-778b8fad4957/volumes"
Oct 31 19:56:59.601118 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:56:59.601103    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" path="/var/lib/kubelet/pods/aba6c1e9-986e-44e5-a460-030a11eacb91/volumes"
Oct 31 19:57:00.048136 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:00.048049    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"]
Oct 31 19:57:00.126672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:00.122466    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d742a2a7-zll46"]
Oct 31 19:57:00.276318 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:00.276259    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:00.523805 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:00.523767    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:00.738352 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:00.738300    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:01.070238 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:01.070198    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:01.419567 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:01.419519    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:01.599797 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:01.599765    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd7871e-1471-4a10-a826-3e3839e08070" path="/var/lib/kubelet/pods/4dd7871e-1471-4a10-a826-3e3839e08070/volumes"
Oct 31 19:57:04.137828 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:04.137783    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:04.355881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:04.355843    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:04.572319 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:04.572217    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:04.918312 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:04.918246    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:05.478083 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:57:05.477945    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:57:05.548674 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:05.544285    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:06.068143 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:06.068106    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:07.602235 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:07.602203    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:11.367194 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:11.367164    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:11.618338 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:11.618236    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:13.274043 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:13.274005    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:14.728028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:14.727997    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:15.005732 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:14.999739    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:19.043953 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:19.043918    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:23.795066 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:23.795032    5006 scope.go:122] "RemoveContainer" containerID="e8a56c087038cc9d21abf9649c0c59b5d1f7f3c65d6fb5af33adbb26c62048ee"
Oct 31 19:57:23.808217 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:23.808195    5006 scope.go:122] "RemoveContainer" containerID="0f468ecec751b8561e4a750592261d2e1aeb94e1802d6b3b550002abd27bc00c"
Oct 31 19:57:23.818574 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:23.818549    5006 scope.go:122] "RemoveContainer" containerID="b3b1cc74b6ddf59d6fa9dcaec5ef246a68dd084ddd8184be0ac30cea58b694ff"
Oct 31 19:57:23.828453 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:23.828432    5006 scope.go:122] "RemoveContainer" containerID="ede1f603a77bd5e561cbe8eeb1adc20878fd514922905884b2e44e1d07e01f89"
Oct 31 19:57:24.765774 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:24.765738    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:24.986761 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:24.986728    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:25.201930 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:25.201895    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:25.525624 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:25.525479    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:29.896529 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:29.896491    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:34.743737 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:34.741158    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:35.394846 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:57:35.394810    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:57:39.683464 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:39.683408    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:44.266541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:44.266498    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:57:44.636974 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:44.636924    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:49.500205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:49.500165    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:54.561332 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:54.561295    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:56.202919 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:56.202879    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:59.804250 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:57:59.804194    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:01.310097 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:01.310063    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:01.932305 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:01.932264    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:02.171601 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:02.171564    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:04.229369 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:04.229332    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:04.447156 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:04.447124    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:04.819662 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:04.819579    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:05.396425 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:58:05.396375    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:58:06.759235 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:06.759201    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:06.992125 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:06.992093    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:10.030866 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:10.030618    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:10.258120 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:10.258081    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:15.616392 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:15.616347    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:19.574838 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:19.574786    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:20.296165 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:20.296120    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:20.669681 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:20.669630    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:20.897244 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:20.897205    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:21.133672 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:21.133140    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:21.362414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:21.362378    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:23.884123 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:23.884083    5006 scope.go:122] "RemoveContainer" containerID="60527bf65342a09d780597f9d9b4e1aa62175390006fb6763487eb0eac8cca20"
Oct 31 19:58:23.979837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:23.979790    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:58:25.888577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:25.888544    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:26.108342 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:26.108305    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:26.453558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:26.453518    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:26.708302 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:26.708205    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:27.598964 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:27.598918    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:30.013748 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:30.013712    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:30.243167 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:30.243098    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:31.368532 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:31.368207    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:31.586899 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:31.586866    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:31.871133 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:31.871096    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:35.396204 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:58:35.396163    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:58:36.756438 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:36.756408    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:38.290986 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:38.290945    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:41.578816 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:41.578782    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:46.291673 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:46.289700    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:46.507086 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:46.507047    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:46.723351 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:46.723323    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:51.385949 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:51.385907    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:56.393501 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:58:56.393463    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:01.161665 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:01.161387    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:03.225506 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.225464    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"]
Oct 31 19:59:03.226174 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.226158    5006 cpu_manager.go:414] "containerMap: removing container" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy"
Oct 31 19:59:03.226226 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.226175    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy"
Oct 31 19:59:03.226265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.226224    5006 cpu_manager.go:414] "containerMap: removing container" podUID="057c2dc1-46d2-49f5-b193-81139d65d2d0" containerName="revoke-keys"
Oct 31 19:59:03.226265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.226230    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057c2dc1-46d2-49f5-b193-81139d65d2d0" containerName="revoke-keys"
Oct 31 19:59:03.226265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.226250    5006 cpu_manager.go:414] "containerMap: removing container" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy"
Oct 31 19:59:03.226265 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.226256    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy"
Oct 31 19:59:03.226402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.226351    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="057c2dc1-46d2-49f5-b193-81139d65d2d0" containerName="revoke-keys"
Oct 31 19:59:03.226402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.226362    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba6c1e9-986e-44e5-a460-030a11eacb91" containerName="istio-proxy"
Oct 31 19:59:03.226402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.226371    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="007305fa-ae8e-4596-9dde-778b8fad4957" containerName="istio-proxy"
Oct 31 19:59:03.230459 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.230440    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.282028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.281986    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"]
Oct 31 19:59:03.328798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.328761    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:03.328996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.328797    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:59:03.328996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.328813    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:59:03.328996 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.328859    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:59:03.342277 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.342239    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvncm\" (UniqueName: \"kubernetes.io/projected/652397b0-4c7d-49fe-aab5-fc67a4a9046f-kube-api-access-zvncm\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.342482 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.342310    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.342546 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.342490    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.443633 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.443592    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvncm\" (UniqueName: \"kubernetes.io/projected/652397b0-4c7d-49fe-aab5-fc67a4a9046f-kube-api-access-zvncm\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.443837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.443733    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.443905 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.443888    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.444413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.444385    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.446283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.446260    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.501642 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.501565    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvncm\" (UniqueName: \"kubernetes.io/projected/652397b0-4c7d-49fe-aab5-fc67a4a9046f-kube-api-access-zvncm\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.545462 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.545423    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:03.919965 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:59:03.919913    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652397b0_4c7d_49fe_aab5_fc67a4a9046f.slice/crio-c19fd49cb28b4fa8484655c80da8b201560f91813d6123c48db0d0a6a19963a8 WatchSource:0}: Error finding container c19fd49cb28b4fa8484655c80da8b201560f91813d6123c48db0d0a6a19963a8: Status 404 returned error can't find the container with id c19fd49cb28b4fa8484655c80da8b201560f91813d6123c48db0d0a6a19963a8
Oct 31 19:59:03.922970 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:03.922879    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"]
Oct 31 19:59:04.004414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:04.004382    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x" event={"ID":"652397b0-4c7d-49fe-aab5-fc67a4a9046f","Type":"ContainerStarted","Data":"c19fd49cb28b4fa8484655c80da8b201560f91813d6123c48db0d0a6a19963a8"}
Oct 31 19:59:05.013835 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:05.013795    5006 generic.go:356] "Generic (PLEG): container finished" podID="652397b0-4c7d-49fe-aab5-fc67a4a9046f" containerID="0cb8f07df5228efbe224e80c4035f52612a66e4722446fd576baba64a2484b04" exitCode=0
Oct 31 19:59:05.014240 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:05.013853    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x" event={"ID":"652397b0-4c7d-49fe-aab5-fc67a4a9046f","Type":"ContainerDied","Data":"0cb8f07df5228efbe224e80c4035f52612a66e4722446fd576baba64a2484b04"}
Oct 31 19:59:05.395475 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:59:05.395436    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:59:06.212549 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.212516    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:06.295985 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.295867    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-xks-ca\") pod \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") "
Oct 31 19:59:06.295985 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.295939    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652397b0-4c7d-49fe-aab5-fc67a4a9046f-kube-api-access-zvncm\" (UniqueName: \"kubernetes.io/projected/652397b0-4c7d-49fe-aab5-fc67a4a9046f-kube-api-access-zvncm\") pod \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") "
Oct 31 19:59:06.296253 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.296101    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-openshift-ca\") pod \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\" (UID: \"652397b0-4c7d-49fe-aab5-fc67a4a9046f\") "
Oct 31 19:59:06.296529 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.296496    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-openshift-ca" pod "652397b0-4c7d-49fe-aab5-fc67a4a9046f" (UID: "652397b0-4c7d-49fe-aab5-fc67a4a9046f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:06.296972 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.296946    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:06.298415 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.298389    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-xks-ca" pod "652397b0-4c7d-49fe-aab5-fc67a4a9046f" (UID: "652397b0-4c7d-49fe-aab5-fc67a4a9046f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:06.298524 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.298412    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652397b0-4c7d-49fe-aab5-fc67a4a9046f-kube-api-access-zvncm" pod "652397b0-4c7d-49fe-aab5-fc67a4a9046f" (UID: "652397b0-4c7d-49fe-aab5-fc67a4a9046f"). InnerVolumeSpecName "kube-api-access-zvncm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:06.398180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.398142    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652397b0-4c7d-49fe-aab5-fc67a4a9046f-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:06.398180 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.398178    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvncm\" (UniqueName: \"kubernetes.io/projected/652397b0-4c7d-49fe-aab5-fc67a4a9046f-kube-api-access-zvncm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:06.566756 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:06.563606    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:07.032260 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.032230    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"
Oct 31 19:59:07.032456 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.032261    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x" event={"ID":"652397b0-4c7d-49fe-aab5-fc67a4a9046f","Type":"ContainerDied","Data":"c19fd49cb28b4fa8484655c80da8b201560f91813d6123c48db0d0a6a19963a8"}
Oct 31 19:59:07.032456 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.032335    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c19fd49cb28b4fa8484655c80da8b201560f91813d6123c48db0d0a6a19963a8"
Oct 31 19:59:07.761214 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.761175    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"]
Oct 31 19:59:07.761942 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.761923    5006 cpu_manager.go:414] "containerMap: removing container" podUID="652397b0-4c7d-49fe-aab5-fc67a4a9046f" containerName="revoke-keys"
Oct 31 19:59:07.762012 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.761943    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652397b0-4c7d-49fe-aab5-fc67a4a9046f" containerName="revoke-keys"
Oct 31 19:59:07.762086 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.762077    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="652397b0-4c7d-49fe-aab5-fc67a4a9046f" containerName="revoke-keys"
Oct 31 19:59:07.766099 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.766077    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:07.817904 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.817868    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"]
Oct 31 19:59:07.818554 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.818532    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:07.863951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.863900    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:59:07.863951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.863939    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:59:07.863951 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.863947    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:59:07.920726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.920689    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp52c\" (UniqueName: \"kubernetes.io/projected/2fcb3780-5dd3-46cc-83ad-2107c71256a9-kube-api-access-cp52c\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:07.920903 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.920745    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:07.920903 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:07.920822    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:08.022420 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.022341    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp52c\" (UniqueName: \"kubernetes.io/projected/2fcb3780-5dd3-46cc-83ad-2107c71256a9-kube-api-access-cp52c\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:08.022420 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.022405    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:08.022620 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.022430    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:08.023073 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.023051    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:08.024923 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.024900    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:08.082410 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.082370    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp52c\" (UniqueName: \"kubernetes.io/projected/2fcb3780-5dd3-46cc-83ad-2107c71256a9-kube-api-access-cp52c\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:08.085232 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.085203    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:08.402517 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.402478    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"]
Oct 31 19:59:08.402757 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.402713    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy" containerID="cri-o://945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31" gracePeriod=30
Oct 31 19:59:08.452981 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.452946    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"]
Oct 31 19:59:08.453185 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.453161    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy" containerID="cri-o://de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7" gracePeriod=30
Oct 31 19:59:08.502311 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:59:08.502277    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcb3780_5dd3_46cc_83ad_2107c71256a9.slice/crio-8c4817b8a67d04dcb2e32e60298a71a4310a18935e1a673b09309ad60a081134 WatchSource:0}: Error finding container 8c4817b8a67d04dcb2e32e60298a71a4310a18935e1a673b09309ad60a081134: Status 404 returned error can't find the container with id 8c4817b8a67d04dcb2e32e60298a71a4310a18935e1a673b09309ad60a081134
Oct 31 19:59:08.518975 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:08.518447    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"]
Oct 31 19:59:09.049020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:09.048916    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq" event={"ID":"2fcb3780-5dd3-46cc-83ad-2107c71256a9","Type":"ContainerStarted","Data":"ab4690125d9423f1c8a8d243a65ad8893f86aca98c3b5ba8503ad3ff41215d12"}
Oct 31 19:59:09.049020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:09.048979    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq" event={"ID":"2fcb3780-5dd3-46cc-83ad-2107c71256a9","Type":"ContainerStarted","Data":"8c4817b8a67d04dcb2e32e60298a71a4310a18935e1a673b09309ad60a081134"}
Oct 31 19:59:09.164260 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:09.164207    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq" podStartSLOduration=2.164190412 podStartE2EDuration="2.164190412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:09.163461995 +0000 UTC m=+2567.692996049" watchObservedRunningTime="2026-10-31 19:59:09.164190412 +0000 UTC m=+2567.693724467"
Oct 31 19:59:09.892871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:09.892833    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"]
Oct 31 19:59:09.893113 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:09.893083    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy" containerID="cri-o://e671837b2c04cf30b182f4db65aac2c5abc0bbdd4d60158291ff044276a72ea4" gracePeriod=30
Oct 31 19:59:09.949104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:09.948319    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"]
Oct 31 19:59:09.949104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:09.948618    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy" containerID="cri-o://4eff85c45043d92a0e3cf39b4bae291992657214ffe5d1d36be4b7480804d9ea" gracePeriod=30
Oct 31 19:59:10.059283 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:10.059242    5006 generic.go:356] "Generic (PLEG): container finished" podID="2fcb3780-5dd3-46cc-83ad-2107c71256a9" containerID="ab4690125d9423f1c8a8d243a65ad8893f86aca98c3b5ba8503ad3ff41215d12" exitCode=0
Oct 31 19:59:10.059728 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:10.059325    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq" event={"ID":"2fcb3780-5dd3-46cc-83ad-2107c71256a9","Type":"ContainerDied","Data":"ab4690125d9423f1c8a8d243a65ad8893f86aca98c3b5ba8503ad3ff41215d12"}
Oct 31 19:59:11.247742 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.247708    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:11.373850 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.373815    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fcb3780-5dd3-46cc-83ad-2107c71256a9-kube-api-access-cp52c\" (UniqueName: \"kubernetes.io/projected/2fcb3780-5dd3-46cc-83ad-2107c71256a9-kube-api-access-cp52c\") pod \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") "
Oct 31 19:59:11.374073 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.374053    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-openshift-ca\") pod \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") "
Oct 31 19:59:11.374183 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.374164    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-xks-ca\") pod \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\" (UID: \"2fcb3780-5dd3-46cc-83ad-2107c71256a9\") "
Oct 31 19:59:11.374497 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.374440    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-openshift-ca" pod "2fcb3780-5dd3-46cc-83ad-2107c71256a9" (UID: "2fcb3780-5dd3-46cc-83ad-2107c71256a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:11.375340 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.375166    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:11.376993 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.376962    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-xks-ca" pod "2fcb3780-5dd3-46cc-83ad-2107c71256a9" (UID: "2fcb3780-5dd3-46cc-83ad-2107c71256a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:11.377155 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.377131    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcb3780-5dd3-46cc-83ad-2107c71256a9-kube-api-access-cp52c" pod "2fcb3780-5dd3-46cc-83ad-2107c71256a9" (UID: "2fcb3780-5dd3-46cc-83ad-2107c71256a9"). InnerVolumeSpecName "kube-api-access-cp52c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:11.476323 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.476282    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp52c\" (UniqueName: \"kubernetes.io/projected/2fcb3780-5dd3-46cc-83ad-2107c71256a9-kube-api-access-cp52c\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:11.476323 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.476316    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fcb3780-5dd3-46cc-83ad-2107c71256a9-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:11.699301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.699210    5006 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:59:11.699301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:11.699281    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:59:12.046119 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:12.046039    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:12.079738 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:12.079691    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq" event={"ID":"2fcb3780-5dd3-46cc-83ad-2107c71256a9","Type":"ContainerDied","Data":"8c4817b8a67d04dcb2e32e60298a71a4310a18935e1a673b09309ad60a081134"}
Oct 31 19:59:12.079974 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:12.079766    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c4817b8a67d04dcb2e32e60298a71a4310a18935e1a673b09309ad60a081134"
Oct 31 19:59:12.079974 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:12.079841    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"
Oct 31 19:59:13.823832 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:13.823796    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:59:13.827711 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:13.827641    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:59:14.015877 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.015787    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-envoy\") pod \"222bdfa6-5c87-47d5-9091-c60527060166\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") "
Oct 31 19:59:14.016069 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.015911    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-token\" (UniqueName: \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-token\") pod \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") "
Oct 31 19:59:14.016069 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.015957    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-kube-api-access-mjsmv\" (UniqueName: \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-kube-api-access-mjsmv\") pod \"222bdfa6-5c87-47d5-9091-c60527060166\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") "
Oct 31 19:59:14.016069 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016028    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-certs\") pod \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") "
Oct 31 19:59:14.016069 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016055    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-socket\") pod \"222bdfa6-5c87-47d5-9091-c60527060166\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") "
Oct 31 19:59:14.016284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016086    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-podinfo\") pod \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") "
Oct 31 19:59:14.016284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016126    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222bdfa6-5c87-47d5-9091-c60527060166-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222bdfa6-5c87-47d5-9091-c60527060166-istiod-ca-cert\") pod \"222bdfa6-5c87-47d5-9091-c60527060166\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") "
Oct 31 19:59:14.016284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016157    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istiod-ca-cert\") pod \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") "
Oct 31 19:59:14.016284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016185    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-data\") pod \"222bdfa6-5c87-47d5-9091-c60527060166\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") "
Oct 31 19:59:14.016284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016262    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-certs\") pod \"222bdfa6-5c87-47d5-9091-c60527060166\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") "
Oct 31 19:59:14.016550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016298    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-envoy\") pod \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") "
Oct 31 19:59:14.016550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016351    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-socket\") pod \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") "
Oct 31 19:59:14.016550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016388    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-istio-token\" (UniqueName: \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-istio-token\") pod \"222bdfa6-5c87-47d5-9091-c60527060166\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") "
Oct 31 19:59:14.016550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016421    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/222bdfa6-5c87-47d5-9091-c60527060166-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222bdfa6-5c87-47d5-9091-c60527060166-istio-podinfo\") pod \"222bdfa6-5c87-47d5-9091-c60527060166\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") "
Oct 31 19:59:14.016550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016502    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-data" pod "222bdfa6-5c87-47d5-9091-c60527060166" (UID: "222bdfa6-5c87-47d5-9091-c60527060166"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:14.016550 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016511    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-credential-socket\") pod \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") "
Oct 31 19:59:14.016865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016578    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-kube-api-access-6sbdv\" (UniqueName: \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-kube-api-access-6sbdv\") pod \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") "
Oct 31 19:59:14.016865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016715    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-data\") pod \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\" (UID: \"2a278f1d-df5d-40be-9b77-9292e1c8bc55\") "
Oct 31 19:59:14.016865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016743    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-credential-socket" pod "2a278f1d-df5d-40be-9b77-9292e1c8bc55" (UID: "2a278f1d-df5d-40be-9b77-9292e1c8bc55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:14.016865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016752    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-credential-socket\") pod \"222bdfa6-5c87-47d5-9091-c60527060166\" (UID: \"222bdfa6-5c87-47d5-9091-c60527060166\") "
Oct 31 19:59:14.016865 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016807    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-socket" pod "222bdfa6-5c87-47d5-9091-c60527060166" (UID: "222bdfa6-5c87-47d5-9091-c60527060166"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:14.017118 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016640    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222bdfa6-5c87-47d5-9091-c60527060166-istiod-ca-cert" pod "222bdfa6-5c87-47d5-9091-c60527060166" (UID: "222bdfa6-5c87-47d5-9091-c60527060166"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:14.017118 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.016986    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-credential-socket" pod "222bdfa6-5c87-47d5-9091-c60527060166" (UID: "222bdfa6-5c87-47d5-9091-c60527060166"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:14.017218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.017197    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-data" pod "2a278f1d-df5d-40be-9b77-9292e1c8bc55" (UID: "2a278f1d-df5d-40be-9b77-9292e1c8bc55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:14.017426 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.017401    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-certs" pod "222bdfa6-5c87-47d5-9091-c60527060166" (UID: "222bdfa6-5c87-47d5-9091-c60527060166"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:14.017789 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.017407    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-certs" pod "2a278f1d-df5d-40be-9b77-9292e1c8bc55" (UID: "2a278f1d-df5d-40be-9b77-9292e1c8bc55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:14.017789 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.017591    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-socket" pod "2a278f1d-df5d-40be-9b77-9292e1c8bc55" (UID: "2a278f1d-df5d-40be-9b77-9292e1c8bc55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:14.018179 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018159    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.018321 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018304    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.018400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018326    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.018400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018326    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istiod-ca-cert" pod "2a278f1d-df5d-40be-9b77-9292e1c8bc55" (UID: "2a278f1d-df5d-40be-9b77-9292e1c8bc55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:14.018400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018340    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.018400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018355    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.018400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018370    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222bdfa6-5c87-47d5-9091-c60527060166-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.018400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018385    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.018400 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018398    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.018795 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018412    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.018972 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018952    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-token" pod "2a278f1d-df5d-40be-9b77-9292e1c8bc55" (UID: "2a278f1d-df5d-40be-9b77-9292e1c8bc55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:14.019079 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.018995    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-envoy" pod "222bdfa6-5c87-47d5-9091-c60527060166" (UID: "222bdfa6-5c87-47d5-9091-c60527060166"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:14.019490 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.019468    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-kube-api-access-mjsmv" pod "222bdfa6-5c87-47d5-9091-c60527060166" (UID: "222bdfa6-5c87-47d5-9091-c60527060166"). InnerVolumeSpecName "kube-api-access-mjsmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:14.019916 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.019891    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-istio-token" pod "222bdfa6-5c87-47d5-9091-c60527060166" (UID: "222bdfa6-5c87-47d5-9091-c60527060166"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:14.020101 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.020075    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-kube-api-access-6sbdv" pod "2a278f1d-df5d-40be-9b77-9292e1c8bc55" (UID: "2a278f1d-df5d-40be-9b77-9292e1c8bc55"). InnerVolumeSpecName "kube-api-access-6sbdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:14.020319 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.020299    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-podinfo" pod "2a278f1d-df5d-40be-9b77-9292e1c8bc55" (UID: "2a278f1d-df5d-40be-9b77-9292e1c8bc55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:59:14.020932 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.020913    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/222bdfa6-5c87-47d5-9091-c60527060166-istio-podinfo" pod "222bdfa6-5c87-47d5-9091-c60527060166" (UID: "222bdfa6-5c87-47d5-9091-c60527060166"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:59:14.021084 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.021070    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-envoy" pod "2a278f1d-df5d-40be-9b77-9292e1c8bc55" (UID: "2a278f1d-df5d-40be-9b77-9292e1c8bc55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:14.096211 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.096164    5006 generic.go:356] "Generic (PLEG): container finished" podID="222bdfa6-5c87-47d5-9091-c60527060166" containerID="de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7" exitCode=0
Oct 31 19:59:14.096391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.096221    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"
Oct 31 19:59:14.096391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.096243    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" event={"ID":"222bdfa6-5c87-47d5-9091-c60527060166","Type":"ContainerDied","Data":"de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7"}
Oct 31 19:59:14.096391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.096341    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4" event={"ID":"222bdfa6-5c87-47d5-9091-c60527060166","Type":"ContainerDied","Data":"d7966698708ab0ba5adc49a99393b16d9862cfbe86e3de4783d9d0183c693140"}
Oct 31 19:59:14.096391 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.096378    5006 scope.go:122] "RemoveContainer" containerID="de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7"
Oct 31 19:59:14.104802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.099304    5006 generic.go:356] "Generic (PLEG): container finished" podID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerID="945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31" exitCode=0
Oct 31 19:59:14.104802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.099473    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" event={"ID":"2a278f1d-df5d-40be-9b77-9292e1c8bc55","Type":"ContainerDied","Data":"945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31"}
Oct 31 19:59:14.104802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.099523    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq" event={"ID":"2a278f1d-df5d-40be-9b77-9292e1c8bc55","Type":"ContainerDied","Data":"0e06e5920eeb4a0b3ef877bf05d417094757d9efbcdb756e2360319028079367"}
Oct 31 19:59:14.104802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.099605    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"
Oct 31 19:59:14.111499 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.111470    5006 scope.go:122] "RemoveContainer" containerID="de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7"
Oct 31 19:59:14.111961 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:59:14.111932    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7\": container with ID starting with de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7 not found: ID does not exist" containerID="de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7"
Oct 31 19:59:14.112138 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.112104    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7"} err="failed to get container status \"de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7\": rpc error: code = NotFound desc = could not find container \"de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7\": container with ID starting with de9c125abb987fa63d87b343752a1b0454305a8f7fbf499fe1a777f7b174a6d7 not found: ID does not exist"
Oct 31 19:59:14.112233 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.112141    5006 scope.go:122] "RemoveContainer" containerID="945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31"
Oct 31 19:59:14.119802 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.119765    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.119926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.119805    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.119926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.119820    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.119926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.119833    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.119926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.119846    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222bdfa6-5c87-47d5-9091-c60527060166-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.119926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.119860    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sbdv\" (UniqueName: \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-kube-api-access-6sbdv\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.119926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.119875    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222bdfa6-5c87-47d5-9091-c60527060166-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.119926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.119890    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a278f1d-df5d-40be-9b77-9292e1c8bc55-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.119926 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.119905    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjsmv\" (UniqueName: \"kubernetes.io/projected/222bdfa6-5c87-47d5-9091-c60527060166-kube-api-access-mjsmv\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:14.124142 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.124115    5006 scope.go:122] "RemoveContainer" containerID="945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31"
Oct 31 19:59:14.124495 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:59:14.124469    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31\": container with ID starting with 945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31 not found: ID does not exist" containerID="945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31"
Oct 31 19:59:14.124577 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.124504    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31"} err="failed to get container status \"945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31\": rpc error: code = NotFound desc = could not find container \"945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31\": container with ID starting with 945b1f451477fe56f05de1bfff09db0b2c5221ae439e725d912f320c91a9df31 not found: ID does not exist"
Oct 31 19:59:14.271946 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.271851    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"]
Oct 31 19:59:14.275572 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.275539    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-fdkn4"]
Oct 31 19:59:14.384786 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.384746    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"]
Oct 31 19:59:14.388555 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.388511    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-bktcq"]
Oct 31 19:59:14.727393 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.727357    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"]
Oct 31 19:59:14.836590 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:14.836446    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"]
Oct 31 19:59:15.113037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.112996    5006 generic.go:356] "Generic (PLEG): container finished" podID="f2fadbb9-e442-4672-af4e-735d47f90808" containerID="4eff85c45043d92a0e3cf39b4bae291992657214ffe5d1d36be4b7480804d9ea" exitCode=0
Oct 31 19:59:15.113196 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.113097    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" event={"ID":"f2fadbb9-e442-4672-af4e-735d47f90808","Type":"ContainerDied","Data":"4eff85c45043d92a0e3cf39b4bae291992657214ffe5d1d36be4b7480804d9ea"}
Oct 31 19:59:15.114800 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.114763    5006 generic.go:356] "Generic (PLEG): container finished" podID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerID="e671837b2c04cf30b182f4db65aac2c5abc0bbdd4d60158291ff044276a72ea4" exitCode=0
Oct 31 19:59:15.114930 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.114854    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" event={"ID":"36d0d17d-7f38-4238-87b9-733e2ab64858","Type":"ContainerDied","Data":"e671837b2c04cf30b182f4db65aac2c5abc0bbdd4d60158291ff044276a72ea4"}
Oct 31 19:59:15.187671 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.186891    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:15.368318 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.368283    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:59:15.382944 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.382909    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:59:15.406539 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.406499    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:15.547708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547601    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-data\") pod \"f2fadbb9-e442-4672-af4e-735d47f90808\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") "
Oct 31 19:59:15.547708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547662    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-credential-socket\") pod \"36d0d17d-7f38-4238-87b9-733e2ab64858\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") "
Oct 31 19:59:15.547919 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547742    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-data\") pod \"36d0d17d-7f38-4238-87b9-733e2ab64858\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") "
Oct 31 19:59:15.547919 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547766    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2fadbb9-e442-4672-af4e-735d47f90808-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fadbb9-e442-4672-af4e-735d47f90808-istio-podinfo\") pod \"f2fadbb9-e442-4672-af4e-735d47f90808\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") "
Oct 31 19:59:15.547919 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547837    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-envoy\") pod \"36d0d17d-7f38-4238-87b9-733e2ab64858\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") "
Oct 31 19:59:15.547919 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547893    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-kube-api-access-66v4w\" (UniqueName: \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-kube-api-access-66v4w\") pod \"f2fadbb9-e442-4672-af4e-735d47f90808\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") "
Oct 31 19:59:15.548146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547941    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d0d17d-7f38-4238-87b9-733e2ab64858-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d0d17d-7f38-4238-87b9-733e2ab64858-istiod-ca-cert\") pod \"36d0d17d-7f38-4238-87b9-733e2ab64858\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") "
Oct 31 19:59:15.548146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547944    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-data" pod "f2fadbb9-e442-4672-af4e-735d47f90808" (UID: "f2fadbb9-e442-4672-af4e-735d47f90808"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:15.548146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547961    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-credential-socket" pod "36d0d17d-7f38-4238-87b9-733e2ab64858" (UID: "36d0d17d-7f38-4238-87b9-733e2ab64858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:15.548146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547976    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-certs\") pod \"f2fadbb9-e442-4672-af4e-735d47f90808\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") "
Oct 31 19:59:15.548146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.547994    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-data" pod "36d0d17d-7f38-4238-87b9-733e2ab64858" (UID: "36d0d17d-7f38-4238-87b9-733e2ab64858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:15.548146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548018    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fadbb9-e442-4672-af4e-735d47f90808-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fadbb9-e442-4672-af4e-735d47f90808-istiod-ca-cert\") pod \"f2fadbb9-e442-4672-af4e-735d47f90808\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") "
Oct 31 19:59:15.548466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548175    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-certs\") pod \"36d0d17d-7f38-4238-87b9-733e2ab64858\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") "
Oct 31 19:59:15.548466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548232    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-socket\") pod \"36d0d17d-7f38-4238-87b9-733e2ab64858\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") "
Oct 31 19:59:15.548466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548307    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-certs" pod "f2fadbb9-e442-4672-af4e-735d47f90808" (UID: "f2fadbb9-e442-4672-af4e-735d47f90808"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:15.548466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548322    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-credential-socket\") pod \"f2fadbb9-e442-4672-af4e-735d47f90808\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") "
Oct 31 19:59:15.548466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548395    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-token\" (UniqueName: \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-token\") pod \"36d0d17d-7f38-4238-87b9-733e2ab64858\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") "
Oct 31 19:59:15.548466 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548437    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-socket\") pod \"f2fadbb9-e442-4672-af4e-735d47f90808\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") "
Oct 31 19:59:15.548801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548477    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-kube-api-access-dhpwz\" (UniqueName: \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-kube-api-access-dhpwz\") pod \"36d0d17d-7f38-4238-87b9-733e2ab64858\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") "
Oct 31 19:59:15.548801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548511    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-podinfo\") pod \"36d0d17d-7f38-4238-87b9-733e2ab64858\" (UID: \"36d0d17d-7f38-4238-87b9-733e2ab64858\") "
Oct 31 19:59:15.548801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548604    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-envoy\") pod \"f2fadbb9-e442-4672-af4e-735d47f90808\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") "
Oct 31 19:59:15.548801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548714    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-istio-token\" (UniqueName: \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-istio-token\") pod \"f2fadbb9-e442-4672-af4e-735d47f90808\" (UID: \"f2fadbb9-e442-4672-af4e-735d47f90808\") "
Oct 31 19:59:15.548801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548313    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d0d17d-7f38-4238-87b9-733e2ab64858-istiod-ca-cert" pod "36d0d17d-7f38-4238-87b9-733e2ab64858" (UID: "36d0d17d-7f38-4238-87b9-733e2ab64858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:15.548801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548782    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fadbb9-e442-4672-af4e-735d47f90808-istiod-ca-cert" pod "f2fadbb9-e442-4672-af4e-735d47f90808" (UID: "f2fadbb9-e442-4672-af4e-735d47f90808"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:15.549110 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548513    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-socket" pod "36d0d17d-7f38-4238-87b9-733e2ab64858" (UID: "36d0d17d-7f38-4238-87b9-733e2ab64858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:15.549110 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548548    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-certs" pod "36d0d17d-7f38-4238-87b9-733e2ab64858" (UID: "36d0d17d-7f38-4238-87b9-733e2ab64858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:15.549110 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.548778    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-credential-socket" pod "f2fadbb9-e442-4672-af4e-735d47f90808" (UID: "f2fadbb9-e442-4672-af4e-735d47f90808"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:15.549543 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.549512    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d0d17d-7f38-4238-87b9-733e2ab64858-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.549543 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.549540    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.549785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.549563    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fadbb9-e442-4672-af4e-735d47f90808-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.549785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.549579    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.549785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.549595    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.549785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.549609    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.549785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.549625    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.549785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.549641    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.549785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.549671    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.550783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.550756    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-socket" pod "f2fadbb9-e442-4672-af4e-735d47f90808" (UID: "f2fadbb9-e442-4672-af4e-735d47f90808"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:15.550947 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.550917    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-kube-api-access-66v4w" pod "f2fadbb9-e442-4672-af4e-735d47f90808" (UID: "f2fadbb9-e442-4672-af4e-735d47f90808"). InnerVolumeSpecName "kube-api-access-66v4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:15.551020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.550941    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2fadbb9-e442-4672-af4e-735d47f90808-istio-podinfo" pod "f2fadbb9-e442-4672-af4e-735d47f90808" (UID: "f2fadbb9-e442-4672-af4e-735d47f90808"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:59:15.551020 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.550997    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-envoy" pod "36d0d17d-7f38-4238-87b9-733e2ab64858" (UID: "36d0d17d-7f38-4238-87b9-733e2ab64858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:15.551299 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.551280    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-kube-api-access-dhpwz" pod "36d0d17d-7f38-4238-87b9-733e2ab64858" (UID: "36d0d17d-7f38-4238-87b9-733e2ab64858"). InnerVolumeSpecName "kube-api-access-dhpwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:15.552048 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.552025    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-token" pod "36d0d17d-7f38-4238-87b9-733e2ab64858" (UID: "36d0d17d-7f38-4238-87b9-733e2ab64858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:15.552137 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.552057    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-podinfo" pod "36d0d17d-7f38-4238-87b9-733e2ab64858" (UID: "36d0d17d-7f38-4238-87b9-733e2ab64858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:59:15.552250 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.552228    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-envoy" pod "f2fadbb9-e442-4672-af4e-735d47f90808" (UID: "f2fadbb9-e442-4672-af4e-735d47f90808"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:15.552311 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.552265    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-istio-token" pod "f2fadbb9-e442-4672-af4e-735d47f90808" (UID: "f2fadbb9-e442-4672-af4e-735d47f90808"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:15.600264 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.600220    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222bdfa6-5c87-47d5-9091-c60527060166" path="/var/lib/kubelet/pods/222bdfa6-5c87-47d5-9091-c60527060166/volumes"
Oct 31 19:59:15.600888 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.600865    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" path="/var/lib/kubelet/pods/2a278f1d-df5d-40be-9b77-9292e1c8bc55/volumes"
Oct 31 19:59:15.651566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.651518    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.651566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.651567    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.651864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.651583    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhpwz\" (UniqueName: \"kubernetes.io/projected/36d0d17d-7f38-4238-87b9-733e2ab64858-kube-api-access-dhpwz\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.651864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.651599    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.651864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.651612    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fadbb9-e442-4672-af4e-735d47f90808-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.651864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.651626    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.651864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.651639    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fadbb9-e442-4672-af4e-735d47f90808-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.651864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.651672    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d0d17d-7f38-4238-87b9-733e2ab64858-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:15.651864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:15.651686    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66v4w\" (UniqueName: \"kubernetes.io/projected/f2fadbb9-e442-4672-af4e-735d47f90808-kube-api-access-66v4w\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:16.125538 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:16.125508    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"
Oct 31 19:59:16.125538 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:16.125523    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj" event={"ID":"f2fadbb9-e442-4672-af4e-735d47f90808","Type":"ContainerDied","Data":"ae62cad914ba1f01b9315d79063de30c460d895cd15498607d208fd0715dd205"}
Oct 31 19:59:16.126135 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:16.125593    5006 scope.go:122] "RemoveContainer" containerID="4eff85c45043d92a0e3cf39b4bae291992657214ffe5d1d36be4b7480804d9ea"
Oct 31 19:59:16.127284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:16.127263    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"
Oct 31 19:59:16.127384 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:16.127291    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q" event={"ID":"36d0d17d-7f38-4238-87b9-733e2ab64858","Type":"ContainerDied","Data":"10fdbef159ccdb1763fdc5b5179d4860eb8b06edea1ca1111e686d93ddb41218"}
Oct 31 19:59:16.136627 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:16.136604    5006 scope.go:122] "RemoveContainer" containerID="e671837b2c04cf30b182f4db65aac2c5abc0bbdd4d60158291ff044276a72ea4"
Oct 31 19:59:16.299535 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:16.299495    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"]
Oct 31 19:59:16.315837 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:16.315773    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-28fjj"]
Oct 31 19:59:16.413717 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:16.413620    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"]
Oct 31 19:59:16.417450 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:16.417421    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-n592q"]
Oct 31 19:59:17.518508 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:17.516568    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:17.599790 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:17.599748    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" path="/var/lib/kubelet/pods/36d0d17d-7f38-4238-87b9-733e2ab64858/volumes"
Oct 31 19:59:17.600336 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:17.600320    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" path="/var/lib/kubelet/pods/f2fadbb9-e442-4672-af4e-735d47f90808/volumes"
Oct 31 19:59:19.696957 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:19.692255    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:19.728798 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:19.728718    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv" podUID="d53c7a5b-0587-4a65-bc21-4f637e6ec278" containerName="payload-processing" containerID="cri-o://ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180" gracePeriod=25
Oct 31 19:59:19.837611 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:19.837564    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh" podUID="3ba7659e-2773-48f1-a0e3-dda638c0c23a" containerName="payload-pre-processing" containerID="cri-o://16ad8abc7b4745e91b5e6f1a8dfc71a4cc9fab3fd4cb1695ab253384d4ddfbc7" gracePeriod=25
Oct 31 19:59:19.907404 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:19.907366    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:20.130871 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.130841    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:59:20.160696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.160665    5006 generic.go:356] "Generic (PLEG): container finished" podID="d53c7a5b-0587-4a65-bc21-4f637e6ec278" containerID="ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180" exitCode=0
Oct 31 19:59:20.160881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.160736    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"
Oct 31 19:59:20.160881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.160740    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv" event={"ID":"d53c7a5b-0587-4a65-bc21-4f637e6ec278","Type":"ContainerDied","Data":"ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180"}
Oct 31 19:59:20.160881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.160796    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv" event={"ID":"d53c7a5b-0587-4a65-bc21-4f637e6ec278","Type":"ContainerDied","Data":"a41f9df595f872f518a278a89a24ec590161297bd88120f95860c16c165e32a2"}
Oct 31 19:59:20.160881 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.160825    5006 scope.go:122] "RemoveContainer" containerID="ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180"
Oct 31 19:59:20.162393 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.162370    5006 generic.go:356] "Generic (PLEG): container finished" podID="3ba7659e-2773-48f1-a0e3-dda638c0c23a" containerID="16ad8abc7b4745e91b5e6f1a8dfc71a4cc9fab3fd4cb1695ab253384d4ddfbc7" exitCode=0
Oct 31 19:59:20.162489 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.162427    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh" event={"ID":"3ba7659e-2773-48f1-a0e3-dda638c0c23a","Type":"ContainerDied","Data":"16ad8abc7b4745e91b5e6f1a8dfc71a4cc9fab3fd4cb1695ab253384d4ddfbc7"}
Oct 31 19:59:20.162489 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.162465    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh" event={"ID":"3ba7659e-2773-48f1-a0e3-dda638c0c23a","Type":"ContainerDied","Data":"e33e1ddfc7f6f988748b40bdb0a2387095a8e6acccd262898369e6a6688c5afe"}
Oct 31 19:59:20.162489 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.162482    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e33e1ddfc7f6f988748b40bdb0a2387095a8e6acccd262898369e6a6688c5afe"
Oct 31 19:59:20.169047 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.169025    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:59:20.171607 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.171588    5006 scope.go:122] "RemoveContainer" containerID="ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180"
Oct 31 19:59:20.171961 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:59:20.171934    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180\": container with ID starting with ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180 not found: ID does not exist" containerID="ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180"
Oct 31 19:59:20.172059 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.171968    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180"} err="failed to get container status \"ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180\": rpc error: code = NotFound desc = could not find container \"ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180\": container with ID starting with ef81c2a07c52e59ef28a0c96bbb88fef84d74756e93b01c8d638800ce27d5180 not found: ID does not exist"
Oct 31 19:59:20.245402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.245308    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba7659e-2773-48f1-a0e3-dda638c0c23a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7659e-2773-48f1-a0e3-dda638c0c23a-plugins-config-volume\") pod \"3ba7659e-2773-48f1-a0e3-dda638c0c23a\" (UID: \"3ba7659e-2773-48f1-a0e3-dda638c0c23a\") "
Oct 31 19:59:20.245402 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.245353    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d53c7a5b-0587-4a65-bc21-4f637e6ec278-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53c7a5b-0587-4a65-bc21-4f637e6ec278-plugins-config-volume\") pod \"d53c7a5b-0587-4a65-bc21-4f637e6ec278\" (UID: \"d53c7a5b-0587-4a65-bc21-4f637e6ec278\") "
Oct 31 19:59:20.245767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.245416    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d53c7a5b-0587-4a65-bc21-4f637e6ec278-kube-api-access-d2vr5\" (UniqueName: \"kubernetes.io/projected/d53c7a5b-0587-4a65-bc21-4f637e6ec278-kube-api-access-d2vr5\") pod \"d53c7a5b-0587-4a65-bc21-4f637e6ec278\" (UID: \"d53c7a5b-0587-4a65-bc21-4f637e6ec278\") "
Oct 31 19:59:20.245767 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.245451    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba7659e-2773-48f1-a0e3-dda638c0c23a-kube-api-access-94kk9\" (UniqueName: \"kubernetes.io/projected/3ba7659e-2773-48f1-a0e3-dda638c0c23a-kube-api-access-94kk9\") pod \"3ba7659e-2773-48f1-a0e3-dda638c0c23a\" (UID: \"3ba7659e-2773-48f1-a0e3-dda638c0c23a\") "
Oct 31 19:59:20.245883 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.245839    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba7659e-2773-48f1-a0e3-dda638c0c23a-plugins-config-volume" pod "3ba7659e-2773-48f1-a0e3-dda638c0c23a" (UID: "3ba7659e-2773-48f1-a0e3-dda638c0c23a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:20.245941 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.245898    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53c7a5b-0587-4a65-bc21-4f637e6ec278-plugins-config-volume" pod "d53c7a5b-0587-4a65-bc21-4f637e6ec278" (UID: "d53c7a5b-0587-4a65-bc21-4f637e6ec278"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:20.246345 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.246318    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ba7659e-2773-48f1-a0e3-dda638c0c23a-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:20.246562 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.246351    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53c7a5b-0587-4a65-bc21-4f637e6ec278-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:20.248396 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.248360    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba7659e-2773-48f1-a0e3-dda638c0c23a-kube-api-access-94kk9" pod "3ba7659e-2773-48f1-a0e3-dda638c0c23a" (UID: "3ba7659e-2773-48f1-a0e3-dda638c0c23a"). InnerVolumeSpecName "kube-api-access-94kk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:20.250359 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.250334    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53c7a5b-0587-4a65-bc21-4f637e6ec278-kube-api-access-d2vr5" pod "d53c7a5b-0587-4a65-bc21-4f637e6ec278" (UID: "d53c7a5b-0587-4a65-bc21-4f637e6ec278"). InnerVolumeSpecName "kube-api-access-d2vr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:20.347506 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.347461    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vr5\" (UniqueName: \"kubernetes.io/projected/d53c7a5b-0587-4a65-bc21-4f637e6ec278-kube-api-access-d2vr5\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:20.347506 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.347495    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94kk9\" (UniqueName: \"kubernetes.io/projected/3ba7659e-2773-48f1-a0e3-dda638c0c23a-kube-api-access-94kk9\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:20.586531 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.586498    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"]
Oct 31 19:59:20.589740 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:20.589714    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7c0d34-58d7cbf77f-rldxv"]
Oct 31 19:59:21.170807 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:21.170781    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"
Oct 31 19:59:21.340467 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:21.340432    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"]
Oct 31 19:59:21.393093 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:21.393034    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7c0d34-84754b49d7-nqbzh"]
Oct 31 19:59:21.601146 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:21.601114    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba7659e-2773-48f1-a0e3-dda638c0c23a" path="/var/lib/kubelet/pods/3ba7659e-2773-48f1-a0e3-dda638c0c23a/volumes"
Oct 31 19:59:21.601632 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:21.601615    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d53c7a5b-0587-4a65-bc21-4f637e6ec278" path="/var/lib/kubelet/pods/d53c7a5b-0587-4a65-bc21-4f637e6ec278/volumes"
Oct 31 19:59:22.988434 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:22.988396    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:23.895294 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:23.895261    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:23.919470 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:23.919436    5006 scope.go:122] "RemoveContainer" containerID="6f49facf0b5a54b32351938ce28d69e8f780ae1f0131bf5a20b5519a6d190b77"
Oct 31 19:59:24.112217 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:24.112182    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:26.287531 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:26.287492    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:26.501820 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:26.501780    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:27.973447 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:27.973413    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:32.805314 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:32.805270    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:34.939893 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:34.939851    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 19:59:35.396526 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:59:35.396472    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 19:59:37.654900 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:37.654833    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:37.975395 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:37.975312    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"]
Oct 31 19:59:37.975579 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:37.975526    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy" containerID="cri-o://a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918" gracePeriod=30
Oct 31 19:59:39.305919 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:39.305875    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"]
Oct 31 19:59:39.306445 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:39.306137    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy" containerID="cri-o://1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd" gracePeriod=30
Oct 31 19:59:39.424801 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:39.424764    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"]
Oct 31 19:59:39.425042 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:39.425011    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy" containerID="cri-o://2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296" gracePeriod=30
Oct 31 19:59:40.000331 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:40.000299    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:40.214347 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:40.214309    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:41.622683 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:41.622626    5006 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:59:41.623101 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:41.622728    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:59:42.114557 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:42.114519    5006 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:59:42.114744 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:42.114589    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:59:42.469730 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:42.469626    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:42.685344 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:42.685171    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:43.252256 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.252198    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:59:43.305793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.305756    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-envoy\") pod \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") "
Oct 31 19:59:43.305987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.305817    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-credential-socket\") pod \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") "
Oct 31 19:59:43.305987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.305881    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-data\") pod \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") "
Oct 31 19:59:43.305987 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.305931    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-token\" (UniqueName: \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-token\") pod \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") "
Oct 31 19:59:43.306155 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.305985    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-certs\") pod \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") "
Oct 31 19:59:43.306155 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306061    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-podinfo\") pod \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") "
Oct 31 19:59:43.306261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306162    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-socket\") pod \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") "
Oct 31 19:59:43.306261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306173    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-credential-socket" pod "6312c0d6-8021-4810-a98c-04aab3ad1e8e" (UID: "6312c0d6-8021-4810-a98c-04aab3ad1e8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:43.306261 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306185    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-data" pod "6312c0d6-8021-4810-a98c-04aab3ad1e8e" (UID: "6312c0d6-8021-4810-a98c-04aab3ad1e8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:43.306414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306261    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-kube-api-access-h9rl8\" (UniqueName: \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-kube-api-access-h9rl8\") pod \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") "
Oct 31 19:59:43.306414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306303    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-certs" pod "6312c0d6-8021-4810-a98c-04aab3ad1e8e" (UID: "6312c0d6-8021-4810-a98c-04aab3ad1e8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:43.306414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306334    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istiod-ca-cert\") pod \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\" (UID: \"6312c0d6-8021-4810-a98c-04aab3ad1e8e\") "
Oct 31 19:59:43.306414 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306403    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-socket" pod "6312c0d6-8021-4810-a98c-04aab3ad1e8e" (UID: "6312c0d6-8021-4810-a98c-04aab3ad1e8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:43.306876 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306852    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istiod-ca-cert" pod "6312c0d6-8021-4810-a98c-04aab3ad1e8e" (UID: "6312c0d6-8021-4810-a98c-04aab3ad1e8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:43.307006 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306876    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:43.307006 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306896    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:43.307006 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306909    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:43.307006 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.306923    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:43.308280 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.308254    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-podinfo" pod "6312c0d6-8021-4810-a98c-04aab3ad1e8e" (UID: "6312c0d6-8021-4810-a98c-04aab3ad1e8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:59:43.308413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.308393    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-envoy" pod "6312c0d6-8021-4810-a98c-04aab3ad1e8e" (UID: "6312c0d6-8021-4810-a98c-04aab3ad1e8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:43.308589 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.308566    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-token" pod "6312c0d6-8021-4810-a98c-04aab3ad1e8e" (UID: "6312c0d6-8021-4810-a98c-04aab3ad1e8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:43.308727 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.308709    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-kube-api-access-h9rl8" pod "6312c0d6-8021-4810-a98c-04aab3ad1e8e" (UID: "6312c0d6-8021-4810-a98c-04aab3ad1e8e"). InnerVolumeSpecName "kube-api-access-h9rl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:43.346028 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.345997    5006 generic.go:356] "Generic (PLEG): container finished" podID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerID="a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918" exitCode=0
Oct 31 19:59:43.346191 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.346046    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"
Oct 31 19:59:43.346191 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.346083    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" event={"ID":"6312c0d6-8021-4810-a98c-04aab3ad1e8e","Type":"ContainerDied","Data":"a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918"}
Oct 31 19:59:43.346191 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.346163    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7" event={"ID":"6312c0d6-8021-4810-a98c-04aab3ad1e8e","Type":"ContainerDied","Data":"1653072b01fcad9f46fa89fae3f75c8e30238eab6ef8ef3e3310f58619c78c3b"}
Oct 31 19:59:43.346191 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.346186    5006 scope.go:122] "RemoveContainer" containerID="a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918"
Oct 31 19:59:43.357706 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.357631    5006 scope.go:122] "RemoveContainer" containerID="a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918"
Oct 31 19:59:43.358051 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:59:43.358024    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918\": container with ID starting with a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918 not found: ID does not exist" containerID="a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918"
Oct 31 19:59:43.358122 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.358060    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918"} err="failed to get container status \"a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918\": rpc error: code = NotFound desc = could not find container \"a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918\": container with ID starting with a54d1ccf1513f19237704b814ec425f7eb1c46d01cabb53b35db6453a1dfa918 not found: ID does not exist"
Oct 31 19:59:43.408490 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.408449    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9rl8\" (UniqueName: \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-kube-api-access-h9rl8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:43.408490 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.408481    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:43.408490 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.408494    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:43.408490 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.408502    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:43.408793 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.408512    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6312c0d6-8021-4810-a98c-04aab3ad1e8e-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:43.533558 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.533458    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"]
Oct 31 19:59:43.537955 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.537927    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-xxct7"]
Oct 31 19:59:43.599965 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:43.599935    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" path="/var/lib/kubelet/pods/6312c0d6-8021-4810-a98c-04aab3ad1e8e/volumes"
Oct 31 19:59:44.611060 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.611028    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:59:44.714320 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.714287    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:59:44.740306 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740266    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-certs\") pod \"447ea240-4362-41e5-b077-4daed43f4c41\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") "
Oct 31 19:59:44.740513 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740379    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-socket\") pod \"447ea240-4362-41e5-b077-4daed43f4c41\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") "
Oct 31 19:59:44.740513 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740428    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-envoy\") pod \"447ea240-4362-41e5-b077-4daed43f4c41\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") "
Oct 31 19:59:44.740677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740557    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-istio-token\" (UniqueName: \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-istio-token\") pod \"447ea240-4362-41e5-b077-4daed43f4c41\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") "
Oct 31 19:59:44.740677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740626    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-certs" pod "447ea240-4362-41e5-b077-4daed43f4c41" (UID: "447ea240-4362-41e5-b077-4daed43f4c41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:44.740677 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740670    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-credential-socket\") pod \"447ea240-4362-41e5-b077-4daed43f4c41\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") "
Oct 31 19:59:44.740845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740742    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-socket" pod "447ea240-4362-41e5-b077-4daed43f4c41" (UID: "447ea240-4362-41e5-b077-4daed43f4c41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:44.740845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740773    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/447ea240-4362-41e5-b077-4daed43f4c41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447ea240-4362-41e5-b077-4daed43f4c41-istio-podinfo\") pod \"447ea240-4362-41e5-b077-4daed43f4c41\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") "
Oct 31 19:59:44.740845 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740833    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/447ea240-4362-41e5-b077-4daed43f4c41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447ea240-4362-41e5-b077-4daed43f4c41-istiod-ca-cert\") pod \"447ea240-4362-41e5-b077-4daed43f4c41\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") "
Oct 31 19:59:44.741014 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740868    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-data\") pod \"447ea240-4362-41e5-b077-4daed43f4c41\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") "
Oct 31 19:59:44.741014 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740900    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-kube-api-access-mhn8f\" (UniqueName: \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-kube-api-access-mhn8f\") pod \"447ea240-4362-41e5-b077-4daed43f4c41\" (UID: \"447ea240-4362-41e5-b077-4daed43f4c41\") "
Oct 31 19:59:44.741014 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.740956    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-credential-socket" pod "447ea240-4362-41e5-b077-4daed43f4c41" (UID: "447ea240-4362-41e5-b077-4daed43f4c41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:44.741404 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.741333    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-data" pod "447ea240-4362-41e5-b077-4daed43f4c41" (UID: "447ea240-4362-41e5-b077-4daed43f4c41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:44.741404 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.741357    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447ea240-4362-41e5-b077-4daed43f4c41-istiod-ca-cert" pod "447ea240-4362-41e5-b077-4daed43f4c41" (UID: "447ea240-4362-41e5-b077-4daed43f4c41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:44.741589 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.741565    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447ea240-4362-41e5-b077-4daed43f4c41-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.741589 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.741586    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.741743 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.741600    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.741743 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.741615    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.741743 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.741630    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.743412 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.743350    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/447ea240-4362-41e5-b077-4daed43f4c41-istio-podinfo" pod "447ea240-4362-41e5-b077-4daed43f4c41" (UID: "447ea240-4362-41e5-b077-4daed43f4c41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:59:44.743412 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.743377    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-envoy" pod "447ea240-4362-41e5-b077-4daed43f4c41" (UID: "447ea240-4362-41e5-b077-4daed43f4c41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:44.743783 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.743753    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-kube-api-access-mhn8f" pod "447ea240-4362-41e5-b077-4daed43f4c41" (UID: "447ea240-4362-41e5-b077-4daed43f4c41"). InnerVolumeSpecName "kube-api-access-mhn8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:44.745284 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.745249    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-istio-token" pod "447ea240-4362-41e5-b077-4daed43f4c41" (UID: "447ea240-4362-41e5-b077-4daed43f4c41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:44.842549 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.842511    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-envoy\") pod \"e11e81ed-de32-43bf-89d7-3503c50fe558\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") "
Oct 31 19:59:44.842747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.842570    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-podinfo\") pod \"e11e81ed-de32-43bf-89d7-3503c50fe558\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") "
Oct 31 19:59:44.842747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.842710    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-token\" (UniqueName: \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-token\") pod \"e11e81ed-de32-43bf-89d7-3503c50fe558\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") "
Oct 31 19:59:44.842747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.842732    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-kube-api-access-5z86h\" (UniqueName: \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-kube-api-access-5z86h\") pod \"e11e81ed-de32-43bf-89d7-3503c50fe558\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") "
Oct 31 19:59:44.842952 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.842788    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e11e81ed-de32-43bf-89d7-3503c50fe558-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11e81ed-de32-43bf-89d7-3503c50fe558-istiod-ca-cert\") pod \"e11e81ed-de32-43bf-89d7-3503c50fe558\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") "
Oct 31 19:59:44.842952 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.842814    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-socket\") pod \"e11e81ed-de32-43bf-89d7-3503c50fe558\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") "
Oct 31 19:59:44.842952 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.842938    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-certs\") pod \"e11e81ed-de32-43bf-89d7-3503c50fe558\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") "
Oct 31 19:59:44.843132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.842980    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-credential-socket\") pod \"e11e81ed-de32-43bf-89d7-3503c50fe558\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") "
Oct 31 19:59:44.843132 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843045    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-data\") pod \"e11e81ed-de32-43bf-89d7-3503c50fe558\" (UID: \"e11e81ed-de32-43bf-89d7-3503c50fe558\") "
Oct 31 19:59:44.843293 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843232    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11e81ed-de32-43bf-89d7-3503c50fe558-istiod-ca-cert" pod "e11e81ed-de32-43bf-89d7-3503c50fe558" (UID: "e11e81ed-de32-43bf-89d7-3503c50fe558"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:44.843413 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843346    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-certs" pod "e11e81ed-de32-43bf-89d7-3503c50fe558" (UID: "e11e81ed-de32-43bf-89d7-3503c50fe558"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:44.843532 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843418    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-socket" pod "e11e81ed-de32-43bf-89d7-3503c50fe558" (UID: "e11e81ed-de32-43bf-89d7-3503c50fe558"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:44.843532 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843430    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-data" pod "e11e81ed-de32-43bf-89d7-3503c50fe558" (UID: "e11e81ed-de32-43bf-89d7-3503c50fe558"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:44.843702 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843547    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-credential-socket" pod "e11e81ed-de32-43bf-89d7-3503c50fe558" (UID: "e11e81ed-de32-43bf-89d7-3503c50fe558"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:44.843827 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843808    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447ea240-4362-41e5-b077-4daed43f4c41-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.843890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843827    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.843890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843836    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11e81ed-de32-43bf-89d7-3503c50fe558-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.843890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843849    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.843890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843862    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.843890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843876    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447ea240-4362-41e5-b077-4daed43f4c41-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.843890 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843885    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.844212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843894    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.844212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.843904    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhn8f\" (UniqueName: \"kubernetes.io/projected/447ea240-4362-41e5-b077-4daed43f4c41-kube-api-access-mhn8f\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.845599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.845411    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-envoy" pod "e11e81ed-de32-43bf-89d7-3503c50fe558" (UID: "e11e81ed-de32-43bf-89d7-3503c50fe558"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:44.845599 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.845444    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-podinfo" pod "e11e81ed-de32-43bf-89d7-3503c50fe558" (UID: "e11e81ed-de32-43bf-89d7-3503c50fe558"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:59:44.846107 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.846071    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-kube-api-access-5z86h" pod "e11e81ed-de32-43bf-89d7-3503c50fe558" (UID: "e11e81ed-de32-43bf-89d7-3503c50fe558"). InnerVolumeSpecName "kube-api-access-5z86h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:44.846198 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.846135    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-token" pod "e11e81ed-de32-43bf-89d7-3503c50fe558" (UID: "e11e81ed-de32-43bf-89d7-3503c50fe558"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:44.945088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.945053    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.945088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.945081    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.945088 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.945093    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:44.945335 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:44.945104    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z86h\" (UniqueName: \"kubernetes.io/projected/e11e81ed-de32-43bf-89d7-3503c50fe558-kube-api-access-5z86h\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:45.375408 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.375373    5006 generic.go:356] "Generic (PLEG): container finished" podID="447ea240-4362-41e5-b077-4daed43f4c41" containerID="1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd" exitCode=0
Oct 31 19:59:45.375747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.375434    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"
Oct 31 19:59:45.375747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.375493    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" event={"ID":"447ea240-4362-41e5-b077-4daed43f4c41","Type":"ContainerDied","Data":"1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd"}
Oct 31 19:59:45.375747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.375573    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv" event={"ID":"447ea240-4362-41e5-b077-4daed43f4c41","Type":"ContainerDied","Data":"6405dcd8db4f353af766f6bedb322309ede6d807c0686903536aad70bec679ef"}
Oct 31 19:59:45.375747 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.375607    5006 scope.go:122] "RemoveContainer" containerID="1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd"
Oct 31 19:59:45.378411 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.378384    5006 generic.go:356] "Generic (PLEG): container finished" podID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerID="2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296" exitCode=0
Oct 31 19:59:45.378537 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.378444    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" event={"ID":"e11e81ed-de32-43bf-89d7-3503c50fe558","Type":"ContainerDied","Data":"2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296"}
Oct 31 19:59:45.378537 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.378498    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp" event={"ID":"e11e81ed-de32-43bf-89d7-3503c50fe558","Type":"ContainerDied","Data":"215a191e80b7d503ad577f4bdf7e1747ecaa4a1da0bd64fbf1e84f101bc32d04"}
Oct 31 19:59:45.378704 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.378579    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"
Oct 31 19:59:45.392487 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.392461    5006 scope.go:122] "RemoveContainer" containerID="1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd"
Oct 31 19:59:45.392874 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:59:45.392846    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd\": container with ID starting with 1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd not found: ID does not exist" containerID="1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd"
Oct 31 19:59:45.392989 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.392882    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd"} err="failed to get container status \"1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd\": rpc error: code = NotFound desc = could not find container \"1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd\": container with ID starting with 1df3813b790b1c47cf592b198d4591c91733d55a0e3a6eb01c657b88441551bd not found: ID does not exist"
Oct 31 19:59:45.392989 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.392900    5006 scope.go:122] "RemoveContainer" containerID="2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296"
Oct 31 19:59:45.403852 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.403829    5006 scope.go:122] "RemoveContainer" containerID="2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296"
Oct 31 19:59:45.404174 ip-10-0-128-104 kubenswrapper[5006]: E1031 19:59:45.404156    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296\": container with ID starting with 2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296 not found: ID does not exist" containerID="2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296"
Oct 31 19:59:45.404228 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.404184    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296"} err="failed to get container status \"2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296\": rpc error: code = NotFound desc = could not find container \"2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296\": container with ID starting with 2d98f5ab956bb83c73013be129504ef54f9f68e98794213100e138060df90296 not found: ID does not exist"
Oct 31 19:59:45.542840 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.542802    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"]
Oct 31 19:59:45.598789 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.598760    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-mg5gv"]
Oct 31 19:59:45.666696 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.666584    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"]
Oct 31 19:59:45.671864 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:45.671834    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-jwwdp"]
Oct 31 19:59:47.606328 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:47.606294    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="447ea240-4362-41e5-b077-4daed43f4c41" path="/var/lib/kubelet/pods/447ea240-4362-41e5-b077-4daed43f4c41/volumes"
Oct 31 19:59:47.606930 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:47.606907    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" path="/var/lib/kubelet/pods/e11e81ed-de32-43bf-89d7-3503c50fe558/volumes"
Oct 31 19:59:47.634231 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:47.634191    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:50.824494 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.824444    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"]
Oct 31 19:59:50.825057 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825040    5006 cpu_manager.go:414] "containerMap: removing container" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy"
Oct 31 19:59:50.825057 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825057    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825071    5006 cpu_manager.go:414] "containerMap: removing container" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825076    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825086    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2fcb3780-5dd3-46cc-83ad-2107c71256a9" containerName="revoke-keys"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825093    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcb3780-5dd3-46cc-83ad-2107c71256a9" containerName="revoke-keys"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825101    5006 cpu_manager.go:414] "containerMap: removing container" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825106    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825116    5006 cpu_manager.go:414] "containerMap: removing container" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825121    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825135    5006 cpu_manager.go:414] "containerMap: removing container" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825140    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825148    5006 cpu_manager.go:414] "containerMap: removing container" podUID="3ba7659e-2773-48f1-a0e3-dda638c0c23a" containerName="payload-pre-processing"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825152    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba7659e-2773-48f1-a0e3-dda638c0c23a" containerName="payload-pre-processing"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825160    5006 cpu_manager.go:414] "containerMap: removing container" podUID="d53c7a5b-0587-4a65-bc21-4f637e6ec278" containerName="payload-processing"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825165    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53c7a5b-0587-4a65-bc21-4f637e6ec278" containerName="payload-processing"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825185    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825189    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825197    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy"
Oct 31 19:59:50.825212 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825201    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy"
Oct 31 19:59:50.826218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825297    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="447ea240-4362-41e5-b077-4daed43f4c41" containerName="istio-proxy"
Oct 31 19:59:50.826218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825305    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d0d17d-7f38-4238-87b9-733e2ab64858" containerName="istio-proxy"
Oct 31 19:59:50.826218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825311    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcb3780-5dd3-46cc-83ad-2107c71256a9" containerName="revoke-keys"
Oct 31 19:59:50.826218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825317    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6312c0d6-8021-4810-a98c-04aab3ad1e8e" containerName="istio-proxy"
Oct 31 19:59:50.826218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825324    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="222bdfa6-5c87-47d5-9091-c60527060166" containerName="istio-proxy"
Oct 31 19:59:50.826218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825331    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2fadbb9-e442-4672-af4e-735d47f90808" containerName="istio-proxy"
Oct 31 19:59:50.826218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825338    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba7659e-2773-48f1-a0e3-dda638c0c23a" containerName="payload-pre-processing"
Oct 31 19:59:50.826218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825345    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53c7a5b-0587-4a65-bc21-4f637e6ec278" containerName="payload-processing"
Oct 31 19:59:50.826218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825353    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11e81ed-de32-43bf-89d7-3503c50fe558" containerName="istio-proxy"
Oct 31 19:59:50.826218 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.825359    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a278f1d-df5d-40be-9b77-9292e1c8bc55" containerName="istio-proxy"
Oct 31 19:59:50.830210 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.830185    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:50.882026 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.881992    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"]
Oct 31 19:59:50.929147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.929107    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:50.929147 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.929115    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:59:50.929388 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.929194    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:59:50.933037 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.933011    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:59:50.941029 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.940996    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49nx\" (UniqueName: \"kubernetes.io/projected/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-kube-api-access-m49nx\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:50.941200 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.941183    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:50.941272 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:50.941255    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:51.042606 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:51.042568    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49nx\" (UniqueName: \"kubernetes.io/projected/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-kube-api-access-m49nx\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:51.042785 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:51.042758    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:51.042936 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:51.042917    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:51.043541 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:51.043517    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:51.045196 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:51.045173    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:51.099933 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:51.099900    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49nx\" (UniqueName: \"kubernetes.io/projected/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-kube-api-access-m49nx\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:51.147085 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:51.147049    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:51.524578 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:59:51.524540    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1d7bf0_ea3e_4e6c_a0d3_f21bcb6d19bf.slice/crio-bd5ec56aeffc612a2ff130e9882ed68a9cd6f149b9b7d733c90d1bb0873d7477 WatchSource:0}: Error finding container bd5ec56aeffc612a2ff130e9882ed68a9cd6f149b9b7d733c90d1bb0873d7477: Status 404 returned error can't find the container with id bd5ec56aeffc612a2ff130e9882ed68a9cd6f149b9b7d733c90d1bb0873d7477
Oct 31 19:59:51.527800 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:51.527767    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"]
Oct 31 19:59:52.431749 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:52.431711    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn" event={"ID":"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf","Type":"ContainerStarted","Data":"6b552e68eda5a1d5a50ec33ae41dd669030ba511dd69ef4ea0c32fb706907636"}
Oct 31 19:59:52.432175 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:52.431772    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn" event={"ID":"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf","Type":"ContainerStarted","Data":"bd5ec56aeffc612a2ff130e9882ed68a9cd6f149b9b7d733c90d1bb0873d7477"}
Oct 31 19:59:52.545753 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:52.545686    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn" podStartSLOduration=2.5456656239999997 podStartE2EDuration="2.545665624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:52.545168222 +0000 UTC m=+2611.074702275" watchObservedRunningTime="2026-10-31 19:59:52.545665624 +0000 UTC m=+2611.075199677"
Oct 31 19:59:52.650324 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:52.650227    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:53.440738 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:53.440705    5006 generic.go:356] "Generic (PLEG): container finished" podID="dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf" containerID="6b552e68eda5a1d5a50ec33ae41dd669030ba511dd69ef4ea0c32fb706907636" exitCode=0
Oct 31 19:59:53.441152 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:53.440801    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn" event={"ID":"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf","Type":"ContainerDied","Data":"6b552e68eda5a1d5a50ec33ae41dd669030ba511dd69ef4ea0c32fb706907636"}
Oct 31 19:59:54.602313 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:54.602286    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:54.691670 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:54.691618    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-openshift-ca\") pod \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") "
Oct 31 19:59:54.691847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:54.691711    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-xks-ca\") pod \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") "
Oct 31 19:59:54.691847 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:54.691839    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-kube-api-access-m49nx\" (UniqueName: \"kubernetes.io/projected/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-kube-api-access-m49nx\") pod \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\" (UID: \"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf\") "
Oct 31 19:59:54.692018 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:54.691989    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-openshift-ca" pod "dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf" (UID: "dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:54.692638 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:54.692606    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:54.693990 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:54.693962    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-xks-ca" pod "dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf" (UID: "dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:54.694104 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:54.694063    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-kube-api-access-m49nx" pod "dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf" (UID: "dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf"). InnerVolumeSpecName "kube-api-access-m49nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:54.793726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:54.793637    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:54.793726 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:54.793721    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m49nx\" (UniqueName: \"kubernetes.io/projected/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf-kube-api-access-m49nx\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:55.455481 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:55.455445    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"
Oct 31 19:59:55.455708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:55.455474    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn" event={"ID":"dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf","Type":"ContainerDied","Data":"bd5ec56aeffc612a2ff130e9882ed68a9cd6f149b9b7d733c90d1bb0873d7477"}
Oct 31 19:59:55.455708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:55.455561    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd5ec56aeffc612a2ff130e9882ed68a9cd6f149b9b7d733c90d1bb0873d7477"
Oct 31 19:59:57.973205 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:57.973161    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:58.873983 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.873944    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"]
Oct 31 19:59:58.874522 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.874509    5006 cpu_manager.go:414] "containerMap: removing container" podUID="dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf" containerName="revoke-keys"
Oct 31 19:59:58.874580 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.874524    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf" containerName="revoke-keys"
Oct 31 19:59:58.874686 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.874645    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf" containerName="revoke-keys"
Oct 31 19:59:58.882177 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.882145    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:58.933899 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.933858    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"]
Oct 31 19:59:58.967479 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.967445    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:58.967684 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.967520    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chcd2\" (UniqueName: \"kubernetes.io/projected/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-kube-api-access-chcd2\") pod \"maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:58.967745 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.967717    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:58.980566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.980529    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:59:58.980566 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.980548    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:58.981013 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.980586    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:59:58.986301 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:58.986279    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 19:59:59.069255 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:59.069163    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chcd2\" (UniqueName: \"kubernetes.io/projected/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-kube-api-access-chcd2\") pod \"maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:59.069425 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:59.069255    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:59.069489 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:59.069427    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:59.070145 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:59.070117    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:59.071708 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:59.071689    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:59.126918 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:59.126878    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chcd2\" (UniqueName: \"kubernetes.io/projected/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-kube-api-access-chcd2\") pod \"maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:59.197769 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:59.197734    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 19:59:59.570286 ip-10-0-128-104 kubenswrapper[5006]: W1031 19:59:59.570246    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69ee09f_4eed_4ad8_874b_a0d992d43bb5.slice/crio-91cc586b8775588a378d086695b8d47b813ad6a0f7da979aca4392f20de7dfa1 WatchSource:0}: Error finding container 91cc586b8775588a378d086695b8d47b813ad6a0f7da979aca4392f20de7dfa1: Status 404 returned error can't find the container with id 91cc586b8775588a378d086695b8d47b813ad6a0f7da979aca4392f20de7dfa1
Oct 31 19:59:59.572879 ip-10-0-128-104 kubenswrapper[5006]: I1031 19:59:59.572856    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"]
Oct 31 20:00:00.164172 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.164141    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"]
Oct 31 20:00:00.168377 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.168355    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"]
Oct 31 20:00:00.168506 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.168471    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"
Oct 31 20:00:00.173320 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.173292    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"]
Oct 31 20:00:00.173464 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.173402    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"
Oct 31 20:00:00.177449 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.177425    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"]
Oct 31 20:00:00.177584 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.177528    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"
Oct 31 20:00:00.181546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.181524    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"
Oct 31 20:00:00.214372 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.214331    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"]
Oct 31 20:00:00.218762 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.218732    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"]
Oct 31 20:00:00.218907 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.218843    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"
Oct 31 20:00:00.226828 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.226799    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"]
Oct 31 20:00:00.226998 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.226935    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"
Oct 31 20:00:00.231925 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.231889    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"]
Oct 31 20:00:00.232055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.232009    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"
Oct 31 20:00:00.236214 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.236191    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"]
Oct 31 20:00:00.236338 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.236291    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"
Oct 31 20:00:00.243470 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.241976    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"]
Oct 31 20:00:00.243852 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.243826    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"
Oct 31 20:00:00.247248 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.247219    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"]
Oct 31 20:00:00.247394 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.247363    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"
Oct 31 20:00:00.252258 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.252230    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"
Oct 31 20:00:00.264933 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.264901    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"]
Oct 31 20:00:00.270254 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.270226    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"]
Oct 31 20:00:00.270392 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.270329    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"
Oct 31 20:00:00.274531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.274505    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"]
Oct 31 20:00:00.274667 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.274618    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"
Oct 31 20:00:00.276478 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.276451    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"]
Oct 31 20:00:00.280198 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.280170    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"]
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291114    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx5lm\" (UniqueName: \"kubernetes.io/projected/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-kube-api-access-jx5lm\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4\" (UID: \"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291176    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdc5l\" (UniqueName: \"kubernetes.io/projected/a47903d5-953b-44c2-9b07-a34c283a21b4-kube-api-access-hdc5l\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj\" (UID: \"a47903d5-953b-44c2-9b07-a34c283a21b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291232    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4\" (UID: \"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291281    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l26f\" (UniqueName: \"kubernetes.io/projected/3d56232f-527a-4659-9756-99c052e97f1f-kube-api-access-2l26f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c\" (UID: \"3d56232f-527a-4659-9756-99c052e97f1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291345    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47903d5-953b-44c2-9b07-a34c283a21b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj\" (UID: \"a47903d5-953b-44c2-9b07-a34c283a21b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291415    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8beb49e6-eb6d-4398-a6a2-f32f176165a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4\" (UID: \"8beb49e6-eb6d-4398-a6a2-f32f176165a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291449    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d56232f-527a-4659-9756-99c052e97f1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c\" (UID: \"3d56232f-527a-4659-9756-99c052e97f1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291495    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjnt\" (UniqueName: \"kubernetes.io/projected/8beb49e6-eb6d-4398-a6a2-f32f176165a1-kube-api-access-jzjnt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4\" (UID: \"8beb49e6-eb6d-4398-a6a2-f32f176165a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291521    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339c95d8-6136-4386-87aa-6da89376a3a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs\" (UID: \"339c95d8-6136-4386-87aa-6da89376a3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291618    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r27t\" (UniqueName: \"kubernetes.io/projected/1cea76a4-21e8-44b7-b9df-b7f19667afb9-kube-api-access-4r27t\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc\" (UID: \"1cea76a4-21e8-44b7-b9df-b7f19667afb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291646    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk74w\" (UniqueName: \"kubernetes.io/projected/339c95d8-6136-4386-87aa-6da89376a3a2-kube-api-access-bk74w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs\" (UID: \"339c95d8-6136-4386-87aa-6da89376a3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291741    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cea76a4-21e8-44b7-b9df-b7f19667afb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc\" (UID: \"1cea76a4-21e8-44b7-b9df-b7f19667afb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291772    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5\" (UID: \"81f82c2d-61ce-409c-8fb3-dadb3e0efb43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"
Oct 31 20:00:00.293730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.291799    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4kw5\" (UniqueName: \"kubernetes.io/projected/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-kube-api-access-q4kw5\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5\" (UID: \"81f82c2d-61ce-409c-8fb3-dadb3e0efb43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"
Oct 31 20:00:00.313826 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.313734    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"]
Oct 31 20:00:00.318540 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.318500    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"]
Oct 31 20:00:00.322644 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.322617    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"]
Oct 31 20:00:00.358391 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.358349    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"]
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.393608    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc5l\" (UniqueName: \"kubernetes.io/projected/a47903d5-953b-44c2-9b07-a34c283a21b4-kube-api-access-hdc5l\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj\" (UID: \"a47903d5-953b-44c2-9b07-a34c283a21b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.393740    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2r8p\" (UniqueName: \"kubernetes.io/projected/4cc97d1e-ef12-49d3-839c-cb2fb8766213-kube-api-access-s2r8p\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv\" (UID: \"4cc97d1e-ef12-49d3-839c-cb2fb8766213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.393828    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr\" (UID: \"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.393866    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4\" (UID: \"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.393918    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l26f\" (UniqueName: \"kubernetes.io/projected/3d56232f-527a-4659-9756-99c052e97f1f-kube-api-access-2l26f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c\" (UID: \"3d56232f-527a-4659-9756-99c052e97f1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394038    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64add86d-fbd5-4293-8600-34d61126b4ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q\" (UID: \"64add86d-fbd5-4293-8600-34d61126b4ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394080    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43664432-b3d5-46ad-9032-bfd6b5b0d584-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5\" (UID: \"43664432-b3d5-46ad-9032-bfd6b5b0d584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394112    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47903d5-953b-44c2-9b07-a34c283a21b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj\" (UID: \"a47903d5-953b-44c2-9b07-a34c283a21b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394160    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkntw\" (UniqueName: \"kubernetes.io/projected/64add86d-fbd5-4293-8600-34d61126b4ec-kube-api-access-lkntw\") pod \"maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q\" (UID: \"64add86d-fbd5-4293-8600-34d61126b4ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394206    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnq5j\" (UniqueName: \"kubernetes.io/projected/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-kube-api-access-rnq5j\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq\" (UID: \"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394253    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8beb49e6-eb6d-4398-a6a2-f32f176165a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4\" (UID: \"8beb49e6-eb6d-4398-a6a2-f32f176165a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394289    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d56232f-527a-4659-9756-99c052e97f1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c\" (UID: \"3d56232f-527a-4659-9756-99c052e97f1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394315    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2lf\" (UniqueName: \"kubernetes.io/projected/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-kube-api-access-xf2lf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l\" (UID: \"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394359    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq\" (UID: \"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394393    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjnt\" (UniqueName: \"kubernetes.io/projected/8beb49e6-eb6d-4398-a6a2-f32f176165a1-kube-api-access-jzjnt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4\" (UID: \"8beb49e6-eb6d-4398-a6a2-f32f176165a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394421    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339c95d8-6136-4386-87aa-6da89376a3a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs\" (UID: \"339c95d8-6136-4386-87aa-6da89376a3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394523    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62j9f\" (UniqueName: \"kubernetes.io/projected/43664432-b3d5-46ad-9032-bfd6b5b0d584-kube-api-access-62j9f\") pod \"maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5\" (UID: \"43664432-b3d5-46ad-9032-bfd6b5b0d584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394598    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r27t\" (UniqueName: \"kubernetes.io/projected/1cea76a4-21e8-44b7-b9df-b7f19667afb9-kube-api-access-4r27t\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc\" (UID: \"1cea76a4-21e8-44b7-b9df-b7f19667afb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394632    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk74w\" (UniqueName: \"kubernetes.io/projected/339c95d8-6136-4386-87aa-6da89376a3a2-kube-api-access-bk74w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs\" (UID: \"339c95d8-6136-4386-87aa-6da89376a3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394709    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cea76a4-21e8-44b7-b9df-b7f19667afb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc\" (UID: \"1cea76a4-21e8-44b7-b9df-b7f19667afb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394743    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5\" (UID: \"81f82c2d-61ce-409c-8fb3-dadb3e0efb43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394953    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4kw5\" (UniqueName: \"kubernetes.io/projected/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-kube-api-access-q4kw5\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5\" (UID: \"81f82c2d-61ce-409c-8fb3-dadb3e0efb43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.394996    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc97d1e-ef12-49d3-839c-cb2fb8766213-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv\" (UID: \"4cc97d1e-ef12-49d3-839c-cb2fb8766213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.395561    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l\" (UID: \"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.395705    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx5lm\" (UniqueName: \"kubernetes.io/projected/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-kube-api-access-jx5lm\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4\" (UID: \"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"
Oct 31 20:00:00.397966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.395734    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92mw\" (UniqueName: \"kubernetes.io/projected/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-kube-api-access-j92mw\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr\" (UID: \"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"
Oct 31 20:00:00.399770 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.399741    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5\" (UID: \"81f82c2d-61ce-409c-8fb3-dadb3e0efb43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"
Oct 31 20:00:00.399894 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.399869    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4\" (UID: \"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"
Oct 31 20:00:00.410109 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.410054    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cea76a4-21e8-44b7-b9df-b7f19667afb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc\" (UID: \"1cea76a4-21e8-44b7-b9df-b7f19667afb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"
Oct 31 20:00:00.411445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.411415    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47903d5-953b-44c2-9b07-a34c283a21b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj\" (UID: \"a47903d5-953b-44c2-9b07-a34c283a21b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"
Oct 31 20:00:00.412293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.412263    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339c95d8-6136-4386-87aa-6da89376a3a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs\" (UID: \"339c95d8-6136-4386-87aa-6da89376a3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"
Oct 31 20:00:00.413397 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.413361    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d56232f-527a-4659-9756-99c052e97f1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c\" (UID: \"3d56232f-527a-4659-9756-99c052e97f1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"
Oct 31 20:00:00.424056 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.424013    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8beb49e6-eb6d-4398-a6a2-f32f176165a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4\" (UID: \"8beb49e6-eb6d-4398-a6a2-f32f176165a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"
Oct 31 20:00:00.424369 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.424349    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"]
Oct 31 20:00:00.430789 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.430756    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"]
Oct 31 20:00:00.434161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.434127    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"]
Oct 31 20:00:00.437297 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.437265    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"]
Oct 31 20:00:00.447993 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.447958    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"]
Oct 31 20:00:00.455287 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.455256    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdc5l\" (UniqueName: \"kubernetes.io/projected/a47903d5-953b-44c2-9b07-a34c283a21b4-kube-api-access-hdc5l\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj\" (UID: \"a47903d5-953b-44c2-9b07-a34c283a21b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"
Oct 31 20:00:00.470673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.467615    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk74w\" (UniqueName: \"kubernetes.io/projected/339c95d8-6136-4386-87aa-6da89376a3a2-kube-api-access-bk74w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs\" (UID: \"339c95d8-6136-4386-87aa-6da89376a3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"
Oct 31 20:00:00.470673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.467852    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l26f\" (UniqueName: \"kubernetes.io/projected/3d56232f-527a-4659-9756-99c052e97f1f-kube-api-access-2l26f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c\" (UID: \"3d56232f-527a-4659-9756-99c052e97f1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"
Oct 31 20:00:00.470673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.468544    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r27t\" (UniqueName: \"kubernetes.io/projected/1cea76a4-21e8-44b7-b9df-b7f19667afb9-kube-api-access-4r27t\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc\" (UID: \"1cea76a4-21e8-44b7-b9df-b7f19667afb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"
Oct 31 20:00:00.470673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.470423    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjnt\" (UniqueName: \"kubernetes.io/projected/8beb49e6-eb6d-4398-a6a2-f32f176165a1-kube-api-access-jzjnt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4\" (UID: \"8beb49e6-eb6d-4398-a6a2-f32f176165a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"
Oct 31 20:00:00.470673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.470450    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4kw5\" (UniqueName: \"kubernetes.io/projected/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-kube-api-access-q4kw5\") pod \"maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5\" (UID: \"81f82c2d-61ce-409c-8fb3-dadb3e0efb43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"
Oct 31 20:00:00.472849 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.472817    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx5lm\" (UniqueName: \"kubernetes.io/projected/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-kube-api-access-jx5lm\") pod \"maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4\" (UID: \"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"
Oct 31 20:00:00.485771 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.485378    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"
Oct 31 20:00:00.494074 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.493852    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.498318    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"]
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.498622    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l\" (UID: \"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.498689    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92mw\" (UniqueName: \"kubernetes.io/projected/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-kube-api-access-j92mw\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr\" (UID: \"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.498744    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r8p\" (UniqueName: \"kubernetes.io/projected/4cc97d1e-ef12-49d3-839c-cb2fb8766213-kube-api-access-s2r8p\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv\" (UID: \"4cc97d1e-ef12-49d3-839c-cb2fb8766213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.498774    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr\" (UID: \"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.498830    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64add86d-fbd5-4293-8600-34d61126b4ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q\" (UID: \"64add86d-fbd5-4293-8600-34d61126b4ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.498877    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43664432-b3d5-46ad-9032-bfd6b5b0d584-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5\" (UID: \"43664432-b3d5-46ad-9032-bfd6b5b0d584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.498928    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkntw\" (UniqueName: \"kubernetes.io/projected/64add86d-fbd5-4293-8600-34d61126b4ec-kube-api-access-lkntw\") pod \"maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q\" (UID: \"64add86d-fbd5-4293-8600-34d61126b4ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.498969    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnq5j\" (UniqueName: \"kubernetes.io/projected/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-kube-api-access-rnq5j\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq\" (UID: \"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.499021    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2lf\" (UniqueName: \"kubernetes.io/projected/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-kube-api-access-xf2lf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l\" (UID: \"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.499057    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq\" (UID: \"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.499132    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62j9f\" (UniqueName: \"kubernetes.io/projected/43664432-b3d5-46ad-9032-bfd6b5b0d584-kube-api-access-62j9f\") pod \"maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5\" (UID: \"43664432-b3d5-46ad-9032-bfd6b5b0d584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.499246    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc97d1e-ef12-49d3-839c-cb2fb8766213-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv\" (UID: \"4cc97d1e-ef12-49d3-839c-cb2fb8766213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.499482    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l\" (UID: \"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.499644    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64add86d-fbd5-4293-8600-34d61126b4ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q\" (UID: \"64add86d-fbd5-4293-8600-34d61126b4ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.500148    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr\" (UID: \"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.500221    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc97d1e-ef12-49d3-839c-cb2fb8766213-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv\" (UID: \"4cc97d1e-ef12-49d3-839c-cb2fb8766213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.500435    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43664432-b3d5-46ad-9032-bfd6b5b0d584-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5\" (UID: \"43664432-b3d5-46ad-9032-bfd6b5b0d584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.500591    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq\" (UID: \"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.500831    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t" event={"ID":"e69ee09f-4eed-4ad8-874b-a0d992d43bb5","Type":"ContainerStarted","Data":"0e08fc04be9d6050716999e82d8e0d4ee353a85ecd80d5f624956761fa7fd97f"}
Oct 31 20:00:00.502027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.500888    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t" event={"ID":"e69ee09f-4eed-4ad8-874b-a0d992d43bb5","Type":"ContainerStarted","Data":"91cc586b8775588a378d086695b8d47b813ad6a0f7da979aca4392f20de7dfa1"}
Oct 31 20:00:00.521035 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.520953    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"
Oct 31 20:00:00.521205 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.521122    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"
Oct 31 20:00:00.538107 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.537856    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"
Oct 31 20:00:00.548998 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.548737    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"
Oct 31 20:00:00.557448 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.557420    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"
Oct 31 20:00:00.559600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.559549    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2lf\" (UniqueName: \"kubernetes.io/projected/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-kube-api-access-xf2lf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l\" (UID: \"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"
Oct 31 20:00:00.560485 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.560434    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2r8p\" (UniqueName: \"kubernetes.io/projected/4cc97d1e-ef12-49d3-839c-cb2fb8766213-kube-api-access-s2r8p\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv\" (UID: \"4cc97d1e-ef12-49d3-839c-cb2fb8766213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"
Oct 31 20:00:00.561797 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.561762    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnq5j\" (UniqueName: \"kubernetes.io/projected/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-kube-api-access-rnq5j\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq\" (UID: \"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"
Oct 31 20:00:00.563147 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.563121    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92mw\" (UniqueName: \"kubernetes.io/projected/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-kube-api-access-j92mw\") pod \"maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr\" (UID: \"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"
Oct 31 20:00:00.565729 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.565629    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62j9f\" (UniqueName: \"kubernetes.io/projected/43664432-b3d5-46ad-9032-bfd6b5b0d584-kube-api-access-62j9f\") pod \"maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5\" (UID: \"43664432-b3d5-46ad-9032-bfd6b5b0d584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"
Oct 31 20:00:00.566806 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.566786    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"
Oct 31 20:00:00.576939 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.576529    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"
Oct 31 20:00:00.577228 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.577208    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkntw\" (UniqueName: \"kubernetes.io/projected/64add86d-fbd5-4293-8600-34d61126b4ec-kube-api-access-lkntw\") pod \"maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q\" (UID: \"64add86d-fbd5-4293-8600-34d61126b4ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"
Oct 31 20:00:00.592552 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.590001    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"
Oct 31 20:00:00.615842 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.612896    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"
Oct 31 20:00:00.621159 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.620946    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"
Oct 31 20:00:00.634190 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:00.633639    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"
Oct 31 20:00:01.091916 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.091885    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"]
Oct 31 20:00:01.156441 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.156411    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"]
Oct 31 20:00:01.188797 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.188769    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"]
Oct 31 20:00:01.207809 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.207780    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:01.213764 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.213732    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"]
Oct 31 20:00:01.258604 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.258572    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"]
Oct 31 20:00:01.298958 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:00:01.298924    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda3dee4_ae07_418b_b5e3_a1dd2fb312ae.slice/crio-d76ceac1cea21edf41d1df358a70c3b261fdb53fb072c8764a8033b3a51485ee WatchSource:0}: Error finding container d76ceac1cea21edf41d1df358a70c3b261fdb53fb072c8764a8033b3a51485ee: Status 404 returned error can't find the container with id d76ceac1cea21edf41d1df358a70c3b261fdb53fb072c8764a8033b3a51485ee
Oct 31 20:00:01.316940 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.315460    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"]
Oct 31 20:00:01.331029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.331001    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"]
Oct 31 20:00:01.335467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.335404    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"]
Oct 31 20:00:01.361363 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:00:01.355930    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64add86d_fbd5_4293_8600_34d61126b4ec.slice/crio-9b0bd9d6b83c910958f5baedfd6d566abc0b399b4325f61bedf617f7152cf65e WatchSource:0}: Error finding container 9b0bd9d6b83c910958f5baedfd6d566abc0b399b4325f61bedf617f7152cf65e: Status 404 returned error can't find the container with id 9b0bd9d6b83c910958f5baedfd6d566abc0b399b4325f61bedf617f7152cf65e
Oct 31 20:00:01.361363 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.360793    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"]
Oct 31 20:00:01.369730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.369700    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"]
Oct 31 20:00:01.381993 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.381966    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"]
Oct 31 20:00:01.385096 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:00:01.385072    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc97d1e_ef12_49d3_839c_cb2fb8766213.slice/crio-65127f42609aa85508b7a04a825509c34ea7437689f5fa9274047f4a6d89121d WatchSource:0}: Error finding container 65127f42609aa85508b7a04a825509c34ea7437689f5fa9274047f4a6d89121d: Status 404 returned error can't find the container with id 65127f42609aa85508b7a04a825509c34ea7437689f5fa9274047f4a6d89121d
Oct 31 20:00:01.386570 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:00:01.386538    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43664432_b3d5_46ad_9032_bfd6b5b0d584.slice/crio-835dc6af1187e78bf8bfeab493464f355916aa1836f2f97fd42c350d23822582 WatchSource:0}: Error finding container 835dc6af1187e78bf8bfeab493464f355916aa1836f2f97fd42c350d23822582: Status 404 returned error can't find the container with id 835dc6af1187e78bf8bfeab493464f355916aa1836f2f97fd42c350d23822582
Oct 31 20:00:01.407145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.407119    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"]
Oct 31 20:00:01.474116 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.474085    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"]
Oct 31 20:00:01.511210 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.511177    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5" event={"ID":"81f82c2d-61ce-409c-8fb3-dadb3e0efb43","Type":"ContainerStarted","Data":"2980b2a1f7d10d83cc9847eea6fab58e2ad62d21c5bac4ca23daa4a8afa50da8"}
Oct 31 20:00:01.512826 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.512792    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" event={"ID":"339c95d8-6136-4386-87aa-6da89376a3a2","Type":"ContainerStarted","Data":"67724e8a61206811ece953fc601929d061904669b4ffde4cdc3be80a7ced4eab"}
Oct 31 20:00:01.514270 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.514244    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" event={"ID":"3d56232f-527a-4659-9756-99c052e97f1f","Type":"ContainerStarted","Data":"c4b588bd62394dd89d9a1fbc5b11350a857b95f2d0e4cda0b400ad498b12c2ac"}
Oct 31 20:00:01.515702 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.515675    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" event={"ID":"8beb49e6-eb6d-4398-a6a2-f32f176165a1","Type":"ContainerStarted","Data":"8d5bba186d4d5ed1598a5f7c3c65dbd6db26b073ce82a3a353cc17248b5d39a4"}
Oct 31 20:00:01.517583 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.517546    5006 generic.go:356] "Generic (PLEG): container finished" podID="e69ee09f-4eed-4ad8-874b-a0d992d43bb5" containerID="0e08fc04be9d6050716999e82d8e0d4ee353a85ecd80d5f624956761fa7fd97f" exitCode=0
Oct 31 20:00:01.517722 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.517638    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t" event={"ID":"e69ee09f-4eed-4ad8-874b-a0d992d43bb5","Type":"ContainerDied","Data":"0e08fc04be9d6050716999e82d8e0d4ee353a85ecd80d5f624956761fa7fd97f"}
Oct 31 20:00:01.519114 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.519089    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4" event={"ID":"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae","Type":"ContainerStarted","Data":"d76ceac1cea21edf41d1df358a70c3b261fdb53fb072c8764a8033b3a51485ee"}
Oct 31 20:00:01.520509 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.520486    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" event={"ID":"1cea76a4-21e8-44b7-b9df-b7f19667afb9","Type":"ContainerStarted","Data":"684aed584b719dfadee3f89b8261155560bbdac6b3a9140cdf90b8ed886a0aa0"}
Oct 31 20:00:01.521702 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.521673    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" event={"ID":"64add86d-fbd5-4293-8600-34d61126b4ec","Type":"ContainerStarted","Data":"9b0bd9d6b83c910958f5baedfd6d566abc0b399b4325f61bedf617f7152cf65e"}
Oct 31 20:00:01.522825 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.522800    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" event={"ID":"43664432-b3d5-46ad-9032-bfd6b5b0d584","Type":"ContainerStarted","Data":"835dc6af1187e78bf8bfeab493464f355916aa1836f2f97fd42c350d23822582"}
Oct 31 20:00:01.523866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.523835    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" event={"ID":"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8","Type":"ContainerStarted","Data":"09de4c13cf3e850387a6d7de12870858bcab30a923d16987a3847b53fc8cdde9"}
Oct 31 20:00:01.525031 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.524981    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" event={"ID":"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca","Type":"ContainerStarted","Data":"c835e052089e768d0172e13a67b244a09962f1de19b2541f4c66e2624401cd96"}
Oct 31 20:00:01.526115 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.526095    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" event={"ID":"4cc97d1e-ef12-49d3-839c-cb2fb8766213","Type":"ContainerStarted","Data":"65127f42609aa85508b7a04a825509c34ea7437689f5fa9274047f4a6d89121d"}
Oct 31 20:00:01.601290 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:00:01.601244    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c56c53_684b_4c9f_b2b4_4bcfe1e7f95a.slice/crio-54fc1e4f15b99bfb3908609dc91bf0f52144917629533df801270aa5e22107c7 WatchSource:0}: Error finding container 54fc1e4f15b99bfb3908609dc91bf0f52144917629533df801270aa5e22107c7: Status 404 returned error can't find the container with id 54fc1e4f15b99bfb3908609dc91bf0f52144917629533df801270aa5e22107c7
Oct 31 20:00:01.747600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:01.747568    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:02.536089 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.536051    5006 generic.go:356] "Generic (PLEG): container finished" podID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerID="438fd7f729636acdc470955653221a9f211484d89dad54f2290bb1e6cd007eb3" exitCode=60
Oct 31 20:00:02.536506 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.536132    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4" event={"ID":"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae","Type":"ContainerDied","Data":"438fd7f729636acdc470955653221a9f211484d89dad54f2290bb1e6cd007eb3"}
Oct 31 20:00:02.536506 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.536457    5006 scope.go:122] "RemoveContainer" containerID="438fd7f729636acdc470955653221a9f211484d89dad54f2290bb1e6cd007eb3"
Oct 31 20:00:02.538084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.538054    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" event={"ID":"1cea76a4-21e8-44b7-b9df-b7f19667afb9","Type":"ContainerStarted","Data":"76bbc40dc70b6e31a189cd801f4dd95a24822f1184a3544177d603a6bdf4ead6"}
Oct 31 20:00:02.539807 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.539775    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" event={"ID":"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a","Type":"ContainerStarted","Data":"80d8ece27689d48105a7a230d57cf449cd220f2f055b98974a8827bdc216e9d2"}
Oct 31 20:00:02.539920 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.539834    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" event={"ID":"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a","Type":"ContainerStarted","Data":"54fc1e4f15b99bfb3908609dc91bf0f52144917629533df801270aa5e22107c7"}
Oct 31 20:00:02.541603 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.541572    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" event={"ID":"64add86d-fbd5-4293-8600-34d61126b4ec","Type":"ContainerStarted","Data":"75755aa6f53f2f8948fcdbf2596a633cd59c2c296cbe51cfefd81989610866c0"}
Oct 31 20:00:02.543444 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.543422    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" event={"ID":"43664432-b3d5-46ad-9032-bfd6b5b0d584","Type":"ContainerStarted","Data":"72ff6da3d43d03ccc3c880cf375b749d63db4013d0a983a14370c600e02a4214"}
Oct 31 20:00:02.545228 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.545196    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" event={"ID":"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8","Type":"ContainerStarted","Data":"10bede06dc6dedce85f3cfc6594407c80664a6bac76bc0fa4f3ee8df241134c0"}
Oct 31 20:00:02.547230 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.547199    5006 generic.go:356] "Generic (PLEG): container finished" podID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerID="3ad1d04e87ba304a9a9452116d695cd77cc4acfd8defd8415b1423a3cf5fa69d" exitCode=60
Oct 31 20:00:02.547363 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.547339    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" event={"ID":"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca","Type":"ContainerDied","Data":"3ad1d04e87ba304a9a9452116d695cd77cc4acfd8defd8415b1423a3cf5fa69d"}
Oct 31 20:00:02.547607 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.547590    5006 scope.go:122] "RemoveContainer" containerID="3ad1d04e87ba304a9a9452116d695cd77cc4acfd8defd8415b1423a3cf5fa69d"
Oct 31 20:00:02.549940 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.549544    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" event={"ID":"4cc97d1e-ef12-49d3-839c-cb2fb8766213","Type":"ContainerStarted","Data":"65a21e00a075144779c268d490b13c2d071c5be9ca85876090af702cd2694ea9"}
Oct 31 20:00:02.551616 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.551593    5006 generic.go:356] "Generic (PLEG): container finished" podID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerID="ac4767f79757786d002560b15d21daaf9647bbad5b3adf3d5187e3086e7d4fc7" exitCode=60
Oct 31 20:00:02.552752 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.552249    5006 scope.go:122] "RemoveContainer" containerID="ac4767f79757786d002560b15d21daaf9647bbad5b3adf3d5187e3086e7d4fc7"
Oct 31 20:00:02.552752 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.552675    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5" event={"ID":"81f82c2d-61ce-409c-8fb3-dadb3e0efb43","Type":"ContainerDied","Data":"ac4767f79757786d002560b15d21daaf9647bbad5b3adf3d5187e3086e7d4fc7"}
Oct 31 20:00:02.556460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.555917    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" event={"ID":"339c95d8-6136-4386-87aa-6da89376a3a2","Type":"ContainerStarted","Data":"1d0a2987318d49ef50877e55e0d43cb2979cf915de842c4d19f28e54216fb951"}
Oct 31 20:00:02.559509 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.559452    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" event={"ID":"3d56232f-527a-4659-9756-99c052e97f1f","Type":"ContainerStarted","Data":"b03f3cd9c29bd5eebf05528b1f8593800dc2127687d8c5a289f226a15f633fd5"}
Oct 31 20:00:02.561989 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.561951    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" event={"ID":"8beb49e6-eb6d-4398-a6a2-f32f176165a1","Type":"ContainerStarted","Data":"ac900dd5abab56d81f6afe6427ecbdb9e029372513d8284bf74ccddee24de327"}
Oct 31 20:00:02.566565 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.566535    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" event={"ID":"a47903d5-953b-44c2-9b07-a34c283a21b4","Type":"ContainerStarted","Data":"60fd35a7cba2987bb005dcd65819ea45617d7d0a5b75eab2c6fa302474669999"}
Oct 31 20:00:02.566697 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.566591    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" event={"ID":"a47903d5-953b-44c2-9b07-a34c283a21b4","Type":"ContainerStarted","Data":"aa07095f38cf9885d9f1cd9b6f8344176fb07152ee1a7ce367a1cbc19d9a2dc9"}
Oct 31 20:00:02.767173 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.766630    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" podStartSLOduration=2.766609426 podStartE2EDuration="2.766609426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.765567522 +0000 UTC m=+2621.295101577" watchObservedRunningTime="2026-10-31 20:00:02.766609426 +0000 UTC m=+2621.296143480"
Oct 31 20:00:02.813965 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.804433    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:02.923571 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.923497    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" podStartSLOduration=2.923473028 podStartE2EDuration="2.923473028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.891908518 +0000 UTC m=+2621.421442573" watchObservedRunningTime="2026-10-31 20:00:02.923473028 +0000 UTC m=+2621.453007085"
Oct 31 20:00:02.939177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:02.939086    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 20:00:03.010035 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.009979    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" podStartSLOduration=3.009962433 podStartE2EDuration="3.009962433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.005572198 +0000 UTC m=+2621.535106248" watchObservedRunningTime="2026-10-31 20:00:03.009962433 +0000 UTC m=+2621.539496490"
Oct 31 20:00:03.024168 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.024080    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:03.060541 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.060507    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-openshift-ca\") pod \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") "
Oct 31 20:00:03.060787 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.060578    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-kube-api-access-chcd2\" (UniqueName: \"kubernetes.io/projected/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-kube-api-access-chcd2\") pod \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") "
Oct 31 20:00:03.060787 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.060630    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-xks-ca\") pod \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\" (UID: \"e69ee09f-4eed-4ad8-874b-a0d992d43bb5\") "
Oct 31 20:00:03.061001 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.060974    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-openshift-ca" pod "e69ee09f-4eed-4ad8-874b-a0d992d43bb5" (UID: "e69ee09f-4eed-4ad8-874b-a0d992d43bb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:03.061336 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.061307    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:03.097830 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.097443    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-kube-api-access-chcd2" pod "e69ee09f-4eed-4ad8-874b-a0d992d43bb5" (UID: "e69ee09f-4eed-4ad8-874b-a0d992d43bb5"). InnerVolumeSpecName "kube-api-access-chcd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:03.098479 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.098442    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-xks-ca" pod "e69ee09f-4eed-4ad8-874b-a0d992d43bb5" (UID: "e69ee09f-4eed-4ad8-874b-a0d992d43bb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:03.125019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.124960    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" podStartSLOduration=3.124944027 podStartE2EDuration="3.124944027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.120804034 +0000 UTC m=+2621.650338106" watchObservedRunningTime="2026-10-31 20:00:03.124944027 +0000 UTC m=+2621.654478114"
Oct 31 20:00:03.162524 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.162490    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chcd2\" (UniqueName: \"kubernetes.io/projected/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-kube-api-access-chcd2\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:03.162524 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.162524    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69ee09f-4eed-4ad8-874b-a0d992d43bb5-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:03.235181 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.235133    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" podStartSLOduration=3.235119033 podStartE2EDuration="3.235119033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.233923379 +0000 UTC m=+2621.763457453" watchObservedRunningTime="2026-10-31 20:00:03.235119033 +0000 UTC m=+2621.764653086"
Oct 31 20:00:03.358287 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.356129    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" podStartSLOduration=3.3561060449999998 podStartE2EDuration="3.356106045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.355118195 +0000 UTC m=+2621.884652250" watchObservedRunningTime="2026-10-31 20:00:03.356106045 +0000 UTC m=+2621.885640122"
Oct 31 20:00:03.374033 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.373995    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:03.468102 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.468036    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" podStartSLOduration=3.46801922 podStartE2EDuration="3.46801922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.46750588 +0000 UTC m=+2621.997039934" watchObservedRunningTime="2026-10-31 20:00:03.46801922 +0000 UTC m=+2621.997553275"
Oct 31 20:00:03.577825 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.577723    5006 generic.go:356] "Generic (PLEG): container finished" podID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerID="65a21e00a075144779c268d490b13c2d071c5be9ca85876090af702cd2694ea9" exitCode=60
Oct 31 20:00:03.578271 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.577833    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" event={"ID":"4cc97d1e-ef12-49d3-839c-cb2fb8766213","Type":"ContainerDied","Data":"65a21e00a075144779c268d490b13c2d071c5be9ca85876090af702cd2694ea9"}
Oct 31 20:00:03.578271 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.578161    5006 scope.go:122] "RemoveContainer" containerID="65a21e00a075144779c268d490b13c2d071c5be9ca85876090af702cd2694ea9"
Oct 31 20:00:03.580047 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.580015    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5" event={"ID":"81f82c2d-61ce-409c-8fb3-dadb3e0efb43","Type":"ContainerStarted","Data":"328ad9ddec90454827a632b4ac6fe8a79f49204e00a00af9a19ab5dfe2573410"}
Oct 31 20:00:03.582049 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.582022    5006 generic.go:356] "Generic (PLEG): container finished" podID="339c95d8-6136-4386-87aa-6da89376a3a2" containerID="1d0a2987318d49ef50877e55e0d43cb2979cf915de842c4d19f28e54216fb951" exitCode=60
Oct 31 20:00:03.582169 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.582065    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" event={"ID":"339c95d8-6136-4386-87aa-6da89376a3a2","Type":"ContainerDied","Data":"1d0a2987318d49ef50877e55e0d43cb2979cf915de842c4d19f28e54216fb951"}
Oct 31 20:00:03.582470 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.582453    5006 scope.go:122] "RemoveContainer" containerID="1d0a2987318d49ef50877e55e0d43cb2979cf915de842c4d19f28e54216fb951"
Oct 31 20:00:03.584406 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.584369    5006 generic.go:356] "Generic (PLEG): container finished" podID="3d56232f-527a-4659-9756-99c052e97f1f" containerID="b03f3cd9c29bd5eebf05528b1f8593800dc2127687d8c5a289f226a15f633fd5" exitCode=60
Oct 31 20:00:03.584493 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.584450    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" event={"ID":"3d56232f-527a-4659-9756-99c052e97f1f","Type":"ContainerDied","Data":"b03f3cd9c29bd5eebf05528b1f8593800dc2127687d8c5a289f226a15f633fd5"}
Oct 31 20:00:03.584743 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.584727    5006 scope.go:122] "RemoveContainer" containerID="b03f3cd9c29bd5eebf05528b1f8593800dc2127687d8c5a289f226a15f633fd5"
Oct 31 20:00:03.586729 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.586704    5006 generic.go:356] "Generic (PLEG): container finished" podID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerID="ac900dd5abab56d81f6afe6427ecbdb9e029372513d8284bf74ccddee24de327" exitCode=60
Oct 31 20:00:03.586832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.586786    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" event={"ID":"8beb49e6-eb6d-4398-a6a2-f32f176165a1","Type":"ContainerDied","Data":"ac900dd5abab56d81f6afe6427ecbdb9e029372513d8284bf74ccddee24de327"}
Oct 31 20:00:03.586989 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.586971    5006 scope.go:122] "RemoveContainer" containerID="ac900dd5abab56d81f6afe6427ecbdb9e029372513d8284bf74ccddee24de327"
Oct 31 20:00:03.588360 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.588339    5006 generic.go:356] "Generic (PLEG): container finished" podID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerID="60fd35a7cba2987bb005dcd65819ea45617d7d0a5b75eab2c6fa302474669999" exitCode=60
Oct 31 20:00:03.588457 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.588426    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" event={"ID":"a47903d5-953b-44c2-9b07-a34c283a21b4","Type":"ContainerDied","Data":"60fd35a7cba2987bb005dcd65819ea45617d7d0a5b75eab2c6fa302474669999"}
Oct 31 20:00:03.588663 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.588632    5006 scope.go:122] "RemoveContainer" containerID="60fd35a7cba2987bb005dcd65819ea45617d7d0a5b75eab2c6fa302474669999"
Oct 31 20:00:03.589884 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.589870    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"
Oct 31 20:00:03.589985 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.589870    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t" event={"ID":"e69ee09f-4eed-4ad8-874b-a0d992d43bb5","Type":"ContainerDied","Data":"91cc586b8775588a378d086695b8d47b813ad6a0f7da979aca4392f20de7dfa1"}
Oct 31 20:00:03.589985 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.589966    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91cc586b8775588a378d086695b8d47b813ad6a0f7da979aca4392f20de7dfa1"
Oct 31 20:00:03.593321 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.593298    5006 generic.go:356] "Generic (PLEG): container finished" podID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerID="76bbc40dc70b6e31a189cd801f4dd95a24822f1184a3544177d603a6bdf4ead6" exitCode=60
Oct 31 20:00:03.593782 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.593694    5006 scope.go:122] "RemoveContainer" containerID="76bbc40dc70b6e31a189cd801f4dd95a24822f1184a3544177d603a6bdf4ead6"
Oct 31 20:00:03.595900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.595877    5006 generic.go:356] "Generic (PLEG): container finished" podID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerID="80d8ece27689d48105a7a230d57cf449cd220f2f055b98974a8827bdc216e9d2" exitCode=60
Oct 31 20:00:03.596230 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.596212    5006 scope.go:122] "RemoveContainer" containerID="80d8ece27689d48105a7a230d57cf449cd220f2f055b98974a8827bdc216e9d2"
Oct 31 20:00:03.613166 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.613133    5006 generic.go:356] "Generic (PLEG): container finished" podID="64add86d-fbd5-4293-8600-34d61126b4ec" containerID="75755aa6f53f2f8948fcdbf2596a633cd59c2c296cbe51cfefd81989610866c0" exitCode=60
Oct 31 20:00:03.613830 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.613680    5006 scope.go:122] "RemoveContainer" containerID="75755aa6f53f2f8948fcdbf2596a633cd59c2c296cbe51cfefd81989610866c0"
Oct 31 20:00:03.616203 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.616169    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" event={"ID":"1cea76a4-21e8-44b7-b9df-b7f19667afb9","Type":"ContainerDied","Data":"76bbc40dc70b6e31a189cd801f4dd95a24822f1184a3544177d603a6bdf4ead6"}
Oct 31 20:00:03.616358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.616244    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" event={"ID":"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a","Type":"ContainerDied","Data":"80d8ece27689d48105a7a230d57cf449cd220f2f055b98974a8827bdc216e9d2"}
Oct 31 20:00:03.616358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.616291    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" event={"ID":"64add86d-fbd5-4293-8600-34d61126b4ec","Type":"ContainerDied","Data":"75755aa6f53f2f8948fcdbf2596a633cd59c2c296cbe51cfefd81989610866c0"}
Oct 31 20:00:03.617118 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.617091    5006 generic.go:356] "Generic (PLEG): container finished" podID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerID="72ff6da3d43d03ccc3c880cf375b749d63db4013d0a983a14370c600e02a4214" exitCode=60
Oct 31 20:00:03.622333 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.617483    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" event={"ID":"43664432-b3d5-46ad-9032-bfd6b5b0d584","Type":"ContainerDied","Data":"72ff6da3d43d03ccc3c880cf375b749d63db4013d0a983a14370c600e02a4214"}
Oct 31 20:00:03.622503 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.618043    5006 scope.go:122] "RemoveContainer" containerID="72ff6da3d43d03ccc3c880cf375b749d63db4013d0a983a14370c600e02a4214"
Oct 31 20:00:03.639375 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.639228    5006 generic.go:356] "Generic (PLEG): container finished" podID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerID="10bede06dc6dedce85f3cfc6594407c80664a6bac76bc0fa4f3ee8df241134c0" exitCode=60
Oct 31 20:00:03.639375 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.639305    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" event={"ID":"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8","Type":"ContainerDied","Data":"10bede06dc6dedce85f3cfc6594407c80664a6bac76bc0fa4f3ee8df241134c0"}
Oct 31 20:00:03.639612 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.639592    5006 scope.go:122] "RemoveContainer" containerID="10bede06dc6dedce85f3cfc6594407c80664a6bac76bc0fa4f3ee8df241134c0"
Oct 31 20:00:03.642334 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:03.642307    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" event={"ID":"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca","Type":"ContainerStarted","Data":"9ab61bc47bcea0d8c0b1f1ceb36dbf2a9c395abaa18c37e10e98411fecdd1856"}
Oct 31 20:00:04.034302 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.031342    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" podStartSLOduration=4.031319678 podStartE2EDuration="4.031319678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.881920546 +0000 UTC m=+2622.411454599" watchObservedRunningTime="2026-10-31 20:00:04.031319678 +0000 UTC m=+2622.560853731"
Oct 31 20:00:04.066035 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.065211    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" podStartSLOduration=4.065189571 podStartE2EDuration="4.065189571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.997016213 +0000 UTC m=+2622.526550293" watchObservedRunningTime="2026-10-31 20:00:04.065189571 +0000 UTC m=+2622.594723624"
Oct 31 20:00:04.118016 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.117077    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" podStartSLOduration=4.117058322 podStartE2EDuration="4.117058322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.115988235 +0000 UTC m=+2622.645522290" watchObservedRunningTime="2026-10-31 20:00:04.117058322 +0000 UTC m=+2622.646592376"
Oct 31 20:00:04.652407 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.652373    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" event={"ID":"8beb49e6-eb6d-4398-a6a2-f32f176165a1","Type":"ContainerStarted","Data":"89df1042a616063e88f9b24bf087ed56a2b00d2f6f464ca697feda0088a247a5"}
Oct 31 20:00:04.654530 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.654496    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" event={"ID":"a47903d5-953b-44c2-9b07-a34c283a21b4","Type":"ContainerStarted","Data":"ca38786461440ae02a77addf198f4d2115916bbd4bc5c8690a91c6cac18da066"}
Oct 31 20:00:04.656570 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.656498    5006 generic.go:356] "Generic (PLEG): container finished" podID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerID="7dfa0945b2bdd5f796ee6d110ee59d1fc5ef69cf16a91f9f5b5b5066409b250a" exitCode=60
Oct 31 20:00:04.656724 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.656565    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4" event={"ID":"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae","Type":"ContainerDied","Data":"7dfa0945b2bdd5f796ee6d110ee59d1fc5ef69cf16a91f9f5b5b5066409b250a"}
Oct 31 20:00:04.656724 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.656615    5006 scope.go:122] "RemoveContainer" containerID="438fd7f729636acdc470955653221a9f211484d89dad54f2290bb1e6cd007eb3"
Oct 31 20:00:04.656929 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.656906    5006 scope.go:122] "RemoveContainer" containerID="7dfa0945b2bdd5f796ee6d110ee59d1fc5ef69cf16a91f9f5b5b5066409b250a"
Oct 31 20:00:04.657383 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:04.657256    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4_odh-ai-gateway-infra(dda3dee4-ae07-418b-b5e3-a1dd2fb312ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae"
Oct 31 20:00:04.658551 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.658527    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" event={"ID":"1cea76a4-21e8-44b7-b9df-b7f19667afb9","Type":"ContainerStarted","Data":"6ef0773881e5dda0a0855168fa4528b73cb786120f78d2c275ab837d892e5332"}
Oct 31 20:00:04.660361 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.660341    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" event={"ID":"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a","Type":"ContainerStarted","Data":"f78068e31be99db856262448654d59ed02bd78202c80252d56434df14d4c8edf"}
Oct 31 20:00:04.662388 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.662362    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" event={"ID":"64add86d-fbd5-4293-8600-34d61126b4ec","Type":"ContainerStarted","Data":"072923a4d8603d895cb76e9675ecda7683505668c1a9275a9a52219362ea839f"}
Oct 31 20:00:04.664455 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.664421    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" event={"ID":"43664432-b3d5-46ad-9032-bfd6b5b0d584","Type":"ContainerStarted","Data":"4da96495800f526f0e37d7e6ac590147e862fc34c16f831e42bd2ce0e8b860c3"}
Oct 31 20:00:04.666519 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.666495    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" event={"ID":"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8","Type":"ContainerStarted","Data":"562b2921b4e7555be7cba39215f550e0c73299938b157d2997242b9d861e98a3"}
Oct 31 20:00:04.668391 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.668364    5006 generic.go:356] "Generic (PLEG): container finished" podID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerID="9ab61bc47bcea0d8c0b1f1ceb36dbf2a9c395abaa18c37e10e98411fecdd1856" exitCode=60
Oct 31 20:00:04.668513 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.668444    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" event={"ID":"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca","Type":"ContainerDied","Data":"9ab61bc47bcea0d8c0b1f1ceb36dbf2a9c395abaa18c37e10e98411fecdd1856"}
Oct 31 20:00:04.668704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.668645    5006 scope.go:122] "RemoveContainer" containerID="9ab61bc47bcea0d8c0b1f1ceb36dbf2a9c395abaa18c37e10e98411fecdd1856"
Oct 31 20:00:04.668961 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:04.668941    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l_odh-ai-gateway-infra(99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca"
Oct 31 20:00:04.670640 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.670620    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" event={"ID":"4cc97d1e-ef12-49d3-839c-cb2fb8766213","Type":"ContainerStarted","Data":"25b8f401cfdf3bed07048e27c937fd9edbd6ede46a91524ac1591712fc847be3"}
Oct 31 20:00:04.671345 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.671328    5006 scope.go:122] "RemoveContainer" containerID="3ad1d04e87ba304a9a9452116d695cd77cc4acfd8defd8415b1423a3cf5fa69d"
Oct 31 20:00:04.672711 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.672691    5006 generic.go:356] "Generic (PLEG): container finished" podID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerID="328ad9ddec90454827a632b4ac6fe8a79f49204e00a00af9a19ab5dfe2573410" exitCode=60
Oct 31 20:00:04.672787 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.672723    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5" event={"ID":"81f82c2d-61ce-409c-8fb3-dadb3e0efb43","Type":"ContainerDied","Data":"328ad9ddec90454827a632b4ac6fe8a79f49204e00a00af9a19ab5dfe2573410"}
Oct 31 20:00:04.672978 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.672962    5006 scope.go:122] "RemoveContainer" containerID="328ad9ddec90454827a632b4ac6fe8a79f49204e00a00af9a19ab5dfe2573410"
Oct 31 20:00:04.673225 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:04.673208    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5_odh-ai-gateway-infra(81f82c2d-61ce-409c-8fb3-dadb3e0efb43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43"
Oct 31 20:00:04.674569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.674549    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" event={"ID":"339c95d8-6136-4386-87aa-6da89376a3a2","Type":"ContainerStarted","Data":"ab97153a24b4e2b6c9a9084a2cfef1e486ca04c4ccc39802dce5c19fe0cf9f2a"}
Oct 31 20:00:04.676337 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.676313    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" event={"ID":"3d56232f-527a-4659-9756-99c052e97f1f","Type":"ContainerStarted","Data":"c092ac495ee1cc4e9bc1f32e0537c45305d74340535e309d30ace1b2e4a9084d"}
Oct 31 20:00:04.685824 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.685799    5006 scope.go:122] "RemoveContainer" containerID="ac4767f79757786d002560b15d21daaf9647bbad5b3adf3d5187e3086e7d4fc7"
Oct 31 20:00:04.714868 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.714805    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5" podStartSLOduration=4.714787819 podStartE2EDuration="4.714787819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.714218336 +0000 UTC m=+2623.243752389" watchObservedRunningTime="2026-10-31 20:00:04.714787819 +0000 UTC m=+2623.244321872"
Oct 31 20:00:04.862547 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:04.862488    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" podStartSLOduration=4.86247214 podStartE2EDuration="4.86247214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.859054738 +0000 UTC m=+2623.388588792" watchObservedRunningTime="2026-10-31 20:00:04.86247214 +0000 UTC m=+2623.392006192"
Oct 31 20:00:05.396992 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.396954    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:00:05.685478 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.685437    5006 generic.go:356] "Generic (PLEG): container finished" podID="339c95d8-6136-4386-87aa-6da89376a3a2" containerID="ab97153a24b4e2b6c9a9084a2cfef1e486ca04c4ccc39802dce5c19fe0cf9f2a" exitCode=60
Oct 31 20:00:05.686006 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.685529    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" event={"ID":"339c95d8-6136-4386-87aa-6da89376a3a2","Type":"ContainerDied","Data":"ab97153a24b4e2b6c9a9084a2cfef1e486ca04c4ccc39802dce5c19fe0cf9f2a"}
Oct 31 20:00:05.686006 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.685612    5006 scope.go:122] "RemoveContainer" containerID="1d0a2987318d49ef50877e55e0d43cb2979cf915de842c4d19f28e54216fb951"
Oct 31 20:00:05.686006 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.685891    5006 scope.go:122] "RemoveContainer" containerID="ab97153a24b4e2b6c9a9084a2cfef1e486ca04c4ccc39802dce5c19fe0cf9f2a"
Oct 31 20:00:05.686306 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.686277    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs_odh-ai-gateway-infra(339c95d8-6136-4386-87aa-6da89376a3a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" podUID="339c95d8-6136-4386-87aa-6da89376a3a2"
Oct 31 20:00:05.687871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.687850    5006 generic.go:356] "Generic (PLEG): container finished" podID="3d56232f-527a-4659-9756-99c052e97f1f" containerID="c092ac495ee1cc4e9bc1f32e0537c45305d74340535e309d30ace1b2e4a9084d" exitCode=60
Oct 31 20:00:05.687973 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.687900    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" event={"ID":"3d56232f-527a-4659-9756-99c052e97f1f","Type":"ContainerDied","Data":"c092ac495ee1cc4e9bc1f32e0537c45305d74340535e309d30ace1b2e4a9084d"}
Oct 31 20:00:05.688251 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.688234    5006 scope.go:122] "RemoveContainer" containerID="c092ac495ee1cc4e9bc1f32e0537c45305d74340535e309d30ace1b2e4a9084d"
Oct 31 20:00:05.688583 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.688565    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c_odh-ai-gateway-infra(3d56232f-527a-4659-9756-99c052e97f1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" podUID="3d56232f-527a-4659-9756-99c052e97f1f"
Oct 31 20:00:05.689907 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.689888    5006 generic.go:356] "Generic (PLEG): container finished" podID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerID="89df1042a616063e88f9b24bf087ed56a2b00d2f6f464ca697feda0088a247a5" exitCode=60
Oct 31 20:00:05.690005 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.689956    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" event={"ID":"8beb49e6-eb6d-4398-a6a2-f32f176165a1","Type":"ContainerDied","Data":"89df1042a616063e88f9b24bf087ed56a2b00d2f6f464ca697feda0088a247a5"}
Oct 31 20:00:05.690198 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.690184    5006 scope.go:122] "RemoveContainer" containerID="89df1042a616063e88f9b24bf087ed56a2b00d2f6f464ca697feda0088a247a5"
Oct 31 20:00:05.690543 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.690481    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4_odh-ai-gateway-infra(8beb49e6-eb6d-4398-a6a2-f32f176165a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1"
Oct 31 20:00:05.691908 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.691888    5006 generic.go:356] "Generic (PLEG): container finished" podID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerID="ca38786461440ae02a77addf198f4d2115916bbd4bc5c8690a91c6cac18da066" exitCode=60
Oct 31 20:00:05.692007 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.691976    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" event={"ID":"a47903d5-953b-44c2-9b07-a34c283a21b4","Type":"ContainerDied","Data":"ca38786461440ae02a77addf198f4d2115916bbd4bc5c8690a91c6cac18da066"}
Oct 31 20:00:05.692162 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.692146    5006 scope.go:122] "RemoveContainer" containerID="ca38786461440ae02a77addf198f4d2115916bbd4bc5c8690a91c6cac18da066"
Oct 31 20:00:05.692422 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.692399    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj_odh-ai-gateway-infra(a47903d5-953b-44c2-9b07-a34c283a21b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4"
Oct 31 20:00:05.693987 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.693967    5006 scope.go:122] "RemoveContainer" containerID="7dfa0945b2bdd5f796ee6d110ee59d1fc5ef69cf16a91f9f5b5b5066409b250a"
Oct 31 20:00:05.694266 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.694243    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4_odh-ai-gateway-infra(dda3dee4-ae07-418b-b5e3-a1dd2fb312ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae"
Oct 31 20:00:05.696356 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.696329    5006 generic.go:356] "Generic (PLEG): container finished" podID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerID="6ef0773881e5dda0a0855168fa4528b73cb786120f78d2c275ab837d892e5332" exitCode=60
Oct 31 20:00:05.696453 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.696368    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" event={"ID":"1cea76a4-21e8-44b7-b9df-b7f19667afb9","Type":"ContainerDied","Data":"6ef0773881e5dda0a0855168fa4528b73cb786120f78d2c275ab837d892e5332"}
Oct 31 20:00:05.696613 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.696599    5006 scope.go:122] "RemoveContainer" containerID="6ef0773881e5dda0a0855168fa4528b73cb786120f78d2c275ab837d892e5332"
Oct 31 20:00:05.696938 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.696915    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc_odh-ai-gateway-infra(1cea76a4-21e8-44b7-b9df-b7f19667afb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9"
Oct 31 20:00:05.698416 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.698394    5006 generic.go:356] "Generic (PLEG): container finished" podID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerID="f78068e31be99db856262448654d59ed02bd78202c80252d56434df14d4c8edf" exitCode=60
Oct 31 20:00:05.698522 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.698453    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" event={"ID":"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a","Type":"ContainerDied","Data":"f78068e31be99db856262448654d59ed02bd78202c80252d56434df14d4c8edf"}
Oct 31 20:00:05.698946 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.698780    5006 scope.go:122] "RemoveContainer" containerID="f78068e31be99db856262448654d59ed02bd78202c80252d56434df14d4c8edf"
Oct 31 20:00:05.699080 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.699058    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr_odh-ai-gateway-infra(e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a"
Oct 31 20:00:05.700713 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.700667    5006 generic.go:356] "Generic (PLEG): container finished" podID="64add86d-fbd5-4293-8600-34d61126b4ec" containerID="072923a4d8603d895cb76e9675ecda7683505668c1a9275a9a52219362ea839f" exitCode=60
Oct 31 20:00:05.700782 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.700735    5006 scope.go:122] "RemoveContainer" containerID="b03f3cd9c29bd5eebf05528b1f8593800dc2127687d8c5a289f226a15f633fd5"
Oct 31 20:00:05.700782 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.700747    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" event={"ID":"64add86d-fbd5-4293-8600-34d61126b4ec","Type":"ContainerDied","Data":"072923a4d8603d895cb76e9675ecda7683505668c1a9275a9a52219362ea839f"}
Oct 31 20:00:05.701064 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.701047    5006 scope.go:122] "RemoveContainer" containerID="072923a4d8603d895cb76e9675ecda7683505668c1a9275a9a52219362ea839f"
Oct 31 20:00:05.701344 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.701321    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q_odh-ai-gateway-infra(64add86d-fbd5-4293-8600-34d61126b4ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" podUID="64add86d-fbd5-4293-8600-34d61126b4ec"
Oct 31 20:00:05.703055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.703035    5006 generic.go:356] "Generic (PLEG): container finished" podID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerID="4da96495800f526f0e37d7e6ac590147e862fc34c16f831e42bd2ce0e8b860c3" exitCode=60
Oct 31 20:00:05.703137 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.703098    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" event={"ID":"43664432-b3d5-46ad-9032-bfd6b5b0d584","Type":"ContainerDied","Data":"4da96495800f526f0e37d7e6ac590147e862fc34c16f831e42bd2ce0e8b860c3"}
Oct 31 20:00:05.703311 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.703298    5006 scope.go:122] "RemoveContainer" containerID="4da96495800f526f0e37d7e6ac590147e862fc34c16f831e42bd2ce0e8b860c3"
Oct 31 20:00:05.703596 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.703575    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5_odh-ai-gateway-infra(43664432-b3d5-46ad-9032-bfd6b5b0d584)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584"
Oct 31 20:00:05.705387 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.705364    5006 generic.go:356] "Generic (PLEG): container finished" podID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerID="562b2921b4e7555be7cba39215f550e0c73299938b157d2997242b9d861e98a3" exitCode=60
Oct 31 20:00:05.705501 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.705448    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" event={"ID":"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8","Type":"ContainerDied","Data":"562b2921b4e7555be7cba39215f550e0c73299938b157d2997242b9d861e98a3"}
Oct 31 20:00:05.705693 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.705643    5006 scope.go:122] "RemoveContainer" containerID="562b2921b4e7555be7cba39215f550e0c73299938b157d2997242b9d861e98a3"
Oct 31 20:00:05.705935 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.705915    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq_odh-ai-gateway-infra(0e171a20-81b9-4f9b-8b74-96b9b05bf4a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8"
Oct 31 20:00:05.707767 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.707747    5006 scope.go:122] "RemoveContainer" containerID="9ab61bc47bcea0d8c0b1f1ceb36dbf2a9c395abaa18c37e10e98411fecdd1856"
Oct 31 20:00:05.708015 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.707995    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l_odh-ai-gateway-infra(99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca"
Oct 31 20:00:05.710076 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.709813    5006 generic.go:356] "Generic (PLEG): container finished" podID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerID="25b8f401cfdf3bed07048e27c937fd9edbd6ede46a91524ac1591712fc847be3" exitCode=60
Oct 31 20:00:05.710076 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.709926    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" event={"ID":"4cc97d1e-ef12-49d3-839c-cb2fb8766213","Type":"ContainerDied","Data":"25b8f401cfdf3bed07048e27c937fd9edbd6ede46a91524ac1591712fc847be3"}
Oct 31 20:00:05.710488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.710417    5006 scope.go:122] "RemoveContainer" containerID="25b8f401cfdf3bed07048e27c937fd9edbd6ede46a91524ac1591712fc847be3"
Oct 31 20:00:05.710756 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.710730    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv_odh-ai-gateway-infra(4cc97d1e-ef12-49d3-839c-cb2fb8766213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213"
Oct 31 20:00:05.712347 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.712327    5006 scope.go:122] "RemoveContainer" containerID="328ad9ddec90454827a632b4ac6fe8a79f49204e00a00af9a19ab5dfe2573410"
Oct 31 20:00:05.712580 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:05.712560    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5_odh-ai-gateway-infra(81f82c2d-61ce-409c-8fb3-dadb3e0efb43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43"
Oct 31 20:00:05.718033 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.718001    5006 scope.go:122] "RemoveContainer" containerID="ac900dd5abab56d81f6afe6427ecbdb9e029372513d8284bf74ccddee24de327"
Oct 31 20:00:05.733829 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.733802    5006 scope.go:122] "RemoveContainer" containerID="60fd35a7cba2987bb005dcd65819ea45617d7d0a5b75eab2c6fa302474669999"
Oct 31 20:00:05.748900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.748878    5006 scope.go:122] "RemoveContainer" containerID="76bbc40dc70b6e31a189cd801f4dd95a24822f1184a3544177d603a6bdf4ead6"
Oct 31 20:00:05.763150 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.763120    5006 scope.go:122] "RemoveContainer" containerID="80d8ece27689d48105a7a230d57cf449cd220f2f055b98974a8827bdc216e9d2"
Oct 31 20:00:05.777461 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.777437    5006 scope.go:122] "RemoveContainer" containerID="75755aa6f53f2f8948fcdbf2596a633cd59c2c296cbe51cfefd81989610866c0"
Oct 31 20:00:05.799141 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.799098    5006 scope.go:122] "RemoveContainer" containerID="72ff6da3d43d03ccc3c880cf375b749d63db4013d0a983a14370c600e02a4214"
Oct 31 20:00:05.823957 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.823934    5006 scope.go:122] "RemoveContainer" containerID="10bede06dc6dedce85f3cfc6594407c80664a6bac76bc0fa4f3ee8df241134c0"
Oct 31 20:00:05.838316 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:05.838296    5006 scope.go:122] "RemoveContainer" containerID="65a21e00a075144779c268d490b13c2d071c5be9ca85876090af702cd2694ea9"
Oct 31 20:00:06.723749 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:06.723718    5006 scope.go:122] "RemoveContainer" containerID="6ef0773881e5dda0a0855168fa4528b73cb786120f78d2c275ab837d892e5332"
Oct 31 20:00:06.724243 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:06.724121    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc_odh-ai-gateway-infra(1cea76a4-21e8-44b7-b9df-b7f19667afb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9"
Oct 31 20:00:06.725588 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:06.725562    5006 scope.go:122] "RemoveContainer" containerID="f78068e31be99db856262448654d59ed02bd78202c80252d56434df14d4c8edf"
Oct 31 20:00:06.725988 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:06.725952    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr_odh-ai-gateway-infra(e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a"
Oct 31 20:00:06.727404 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:06.727380    5006 scope.go:122] "RemoveContainer" containerID="072923a4d8603d895cb76e9675ecda7683505668c1a9275a9a52219362ea839f"
Oct 31 20:00:06.727763 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:06.727735    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q_odh-ai-gateway-infra(64add86d-fbd5-4293-8600-34d61126b4ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" podUID="64add86d-fbd5-4293-8600-34d61126b4ec"
Oct 31 20:00:06.729474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:06.729306    5006 scope.go:122] "RemoveContainer" containerID="4da96495800f526f0e37d7e6ac590147e862fc34c16f831e42bd2ce0e8b860c3"
Oct 31 20:00:06.729690 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:06.729629    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5_odh-ai-gateway-infra(43664432-b3d5-46ad-9032-bfd6b5b0d584)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584"
Oct 31 20:00:06.731335 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:06.731314    5006 scope.go:122] "RemoveContainer" containerID="562b2921b4e7555be7cba39215f550e0c73299938b157d2997242b9d861e98a3"
Oct 31 20:00:06.731633 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:06.731596    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq_odh-ai-gateway-infra(0e171a20-81b9-4f9b-8b74-96b9b05bf4a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8"
Oct 31 20:00:06.733327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:06.733303    5006 scope.go:122] "RemoveContainer" containerID="25b8f401cfdf3bed07048e27c937fd9edbd6ede46a91524ac1591712fc847be3"
Oct 31 20:00:06.733589 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:06.733569    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv_odh-ai-gateway-infra(4cc97d1e-ef12-49d3-839c-cb2fb8766213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213"
Oct 31 20:00:06.734901 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:06.734883    5006 scope.go:122] "RemoveContainer" containerID="ab97153a24b4e2b6c9a9084a2cfef1e486ca04c4ccc39802dce5c19fe0cf9f2a"
Oct 31 20:00:06.735166 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:06.735142    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs_odh-ai-gateway-infra(339c95d8-6136-4386-87aa-6da89376a3a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" podUID="339c95d8-6136-4386-87aa-6da89376a3a2"
Oct 31 20:00:06.736822 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:06.736801    5006 scope.go:122] "RemoveContainer" containerID="c092ac495ee1cc4e9bc1f32e0537c45305d74340535e309d30ace1b2e4a9084d"
Oct 31 20:00:06.737096 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:06.737063    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c_odh-ai-gateway-infra(3d56232f-527a-4659-9756-99c052e97f1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" podUID="3d56232f-527a-4659-9756-99c052e97f1f"
Oct 31 20:00:06.738749 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:06.738728    5006 scope.go:122] "RemoveContainer" containerID="89df1042a616063e88f9b24bf087ed56a2b00d2f6f464ca697feda0088a247a5"
Oct 31 20:00:06.738965 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:06.738941    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4_odh-ai-gateway-infra(8beb49e6-eb6d-4398-a6a2-f32f176165a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1"
Oct 31 20:00:06.740543 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:06.740525    5006 scope.go:122] "RemoveContainer" containerID="ca38786461440ae02a77addf198f4d2115916bbd4bc5c8690a91c6cac18da066"
Oct 31 20:00:06.740784 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:06.740764    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj_odh-ai-gateway-infra(a47903d5-953b-44c2-9b07-a34c283a21b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4"
Oct 31 20:00:08.744019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:08.743974    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:08.960471 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:08.960437    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:10.407531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.407493    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"]
Oct 31 20:00:10.408246 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.408228    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e69ee09f-4eed-4ad8-874b-a0d992d43bb5" containerName="revoke-keys"
Oct 31 20:00:10.408302 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.408247    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69ee09f-4eed-4ad8-874b-a0d992d43bb5" containerName="revoke-keys"
Oct 31 20:00:10.408392 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.408382    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69ee09f-4eed-4ad8-874b-a0d992d43bb5" containerName="revoke-keys"
Oct 31 20:00:10.413467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.413448    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:10.467122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.467085    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"]
Oct 31 20:00:10.505586 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.505528    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35970881-0106-4496-ad86-d908337d2c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:10.505841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.505785    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35970881-0106-4496-ad86-d908337d2c7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:10.505841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.505820    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dchrn\" (UniqueName: \"kubernetes.io/projected/35970881-0106-4496-ad86-d908337d2c7c-kube-api-access-dchrn\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:10.510790 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.510762    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:00:10.607075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.607027    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35970881-0106-4496-ad86-d908337d2c7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:10.607075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.607075    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dchrn\" (UniqueName: \"kubernetes.io/projected/35970881-0106-4496-ad86-d908337d2c7c-kube-api-access-dchrn\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:10.607304 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.607195    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35970881-0106-4496-ad86-d908337d2c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:10.607908 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.607873    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35970881-0106-4496-ad86-d908337d2c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:10.609523 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.609496    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35970881-0106-4496-ad86-d908337d2c7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:10.664713 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.664627    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dchrn\" (UniqueName: \"kubernetes.io/projected/35970881-0106-4496-ad86-d908337d2c7c-kube-api-access-dchrn\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:10.727973 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:10.727932    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:11.121503 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:00:11.121452    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35970881_0106_4496_ad86_d908337d2c7c.slice/crio-8177c762ae05028ee9aaa22d7ce65b6045f85ffff31edd04418a8ad9f9af8c6e WatchSource:0}: Error finding container 8177c762ae05028ee9aaa22d7ce65b6045f85ffff31edd04418a8ad9f9af8c6e: Status 404 returned error can't find the container with id 8177c762ae05028ee9aaa22d7ce65b6045f85ffff31edd04418a8ad9f9af8c6e
Oct 31 20:00:11.126366 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:11.126333    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"]
Oct 31 20:00:11.913419 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:11.913384    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2" event={"ID":"35970881-0106-4496-ad86-d908337d2c7c","Type":"ContainerStarted","Data":"23ac4c086611a6f69c99f2b0abcc8564b2d00bff6037a15226ce6f846531ba12"}
Oct 31 20:00:11.914219 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:11.914183    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2" event={"ID":"35970881-0106-4496-ad86-d908337d2c7c","Type":"ContainerStarted","Data":"8177c762ae05028ee9aaa22d7ce65b6045f85ffff31edd04418a8ad9f9af8c6e"}
Oct 31 20:00:12.031855 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:12.031795    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2" podStartSLOduration=2.031775276 podStartE2EDuration="2.031775276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:12.027743181 +0000 UTC m=+2630.557277233" watchObservedRunningTime="2026-10-31 20:00:12.031775276 +0000 UTC m=+2630.561309328"
Oct 31 20:00:12.923087 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:12.923049    5006 generic.go:356] "Generic (PLEG): container finished" podID="35970881-0106-4496-ad86-d908337d2c7c" containerID="23ac4c086611a6f69c99f2b0abcc8564b2d00bff6037a15226ce6f846531ba12" exitCode=0
Oct 31 20:00:12.923638 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:12.923173    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2" event={"ID":"35970881-0106-4496-ad86-d908337d2c7c","Type":"ContainerDied","Data":"23ac4c086611a6f69c99f2b0abcc8564b2d00bff6037a15226ce6f846531ba12"}
Oct 31 20:00:14.117106 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.117071    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:14.184779 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.184736    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35970881-0106-4496-ad86-d908337d2c7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35970881-0106-4496-ad86-d908337d2c7c-maas-api-openshift-ca\") pod \"35970881-0106-4496-ad86-d908337d2c7c\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") "
Oct 31 20:00:14.184989 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.184829    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35970881-0106-4496-ad86-d908337d2c7c-kube-api-access-dchrn\" (UniqueName: \"kubernetes.io/projected/35970881-0106-4496-ad86-d908337d2c7c-kube-api-access-dchrn\") pod \"35970881-0106-4496-ad86-d908337d2c7c\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") "
Oct 31 20:00:14.184989 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.184955    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35970881-0106-4496-ad86-d908337d2c7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35970881-0106-4496-ad86-d908337d2c7c-maas-api-xks-ca\") pod \"35970881-0106-4496-ad86-d908337d2c7c\" (UID: \"35970881-0106-4496-ad86-d908337d2c7c\") "
Oct 31 20:00:14.185106 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.185059    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35970881-0106-4496-ad86-d908337d2c7c-maas-api-openshift-ca" pod "35970881-0106-4496-ad86-d908337d2c7c" (UID: "35970881-0106-4496-ad86-d908337d2c7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:14.185645 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.185625    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35970881-0106-4496-ad86-d908337d2c7c-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:14.187172 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.187117    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35970881-0106-4496-ad86-d908337d2c7c-kube-api-access-dchrn" pod "35970881-0106-4496-ad86-d908337d2c7c" (UID: "35970881-0106-4496-ad86-d908337d2c7c"). InnerVolumeSpecName "kube-api-access-dchrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:14.187172 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.187147    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35970881-0106-4496-ad86-d908337d2c7c-maas-api-xks-ca" pod "35970881-0106-4496-ad86-d908337d2c7c" (UID: "35970881-0106-4496-ad86-d908337d2c7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:14.282383 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.282335    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:14.287974 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.287932    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dchrn\" (UniqueName: \"kubernetes.io/projected/35970881-0106-4496-ad86-d908337d2c7c-kube-api-access-dchrn\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:14.287974 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.287975    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35970881-0106-4496-ad86-d908337d2c7c-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:14.498901 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.498812    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:14.948796 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.948761    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"
Oct 31 20:00:14.948980 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.948793    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2" event={"ID":"35970881-0106-4496-ad86-d908337d2c7c","Type":"ContainerDied","Data":"8177c762ae05028ee9aaa22d7ce65b6045f85ffff31edd04418a8ad9f9af8c6e"}
Oct 31 20:00:14.948980 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:14.948849    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8177c762ae05028ee9aaa22d7ce65b6045f85ffff31edd04418a8ad9f9af8c6e"
Oct 31 20:00:15.632674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:15.632569    5006 scope.go:122] "RemoveContainer" containerID="7dfa0945b2bdd5f796ee6d110ee59d1fc5ef69cf16a91f9f5b5b5066409b250a"
Oct 31 20:00:15.633797 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:15.633773    5006 scope.go:122] "RemoveContainer" containerID="072923a4d8603d895cb76e9675ecda7683505668c1a9275a9a52219362ea839f"
Oct 31 20:00:15.634928 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:15.634904    5006 scope.go:122] "RemoveContainer" containerID="ca38786461440ae02a77addf198f4d2115916bbd4bc5c8690a91c6cac18da066"
Oct 31 20:00:16.591276 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.591246    5006 scope.go:122] "RemoveContainer" containerID="328ad9ddec90454827a632b4ac6fe8a79f49204e00a00af9a19ab5dfe2573410"
Oct 31 20:00:16.591448 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.591406    5006 scope.go:122] "RemoveContainer" containerID="25b8f401cfdf3bed07048e27c937fd9edbd6ede46a91524ac1591712fc847be3"
Oct 31 20:00:16.591492 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.591470    5006 scope.go:122] "RemoveContainer" containerID="9ab61bc47bcea0d8c0b1f1ceb36dbf2a9c395abaa18c37e10e98411fecdd1856"
Oct 31 20:00:16.965708 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.965676    5006 generic.go:356] "Generic (PLEG): container finished" podID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerID="f94c2301065ba8c82a9dddee6ad2900532ea3673f38b09463437dac317c02448" exitCode=60
Oct 31 20:00:16.966161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.965751    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4" event={"ID":"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae","Type":"ContainerDied","Data":"f94c2301065ba8c82a9dddee6ad2900532ea3673f38b09463437dac317c02448"}
Oct 31 20:00:16.966161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.965826    5006 scope.go:122] "RemoveContainer" containerID="7dfa0945b2bdd5f796ee6d110ee59d1fc5ef69cf16a91f9f5b5b5066409b250a"
Oct 31 20:00:16.966289 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.966259    5006 scope.go:122] "RemoveContainer" containerID="f94c2301065ba8c82a9dddee6ad2900532ea3673f38b09463437dac317c02448"
Oct 31 20:00:16.966703 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:16.966640    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4_odh-ai-gateway-infra(dda3dee4-ae07-418b-b5e3-a1dd2fb312ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae"
Oct 31 20:00:16.968018 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.967995    5006 generic.go:356] "Generic (PLEG): container finished" podID="64add86d-fbd5-4293-8600-34d61126b4ec" containerID="85123cab1866c7d571f7c6767920d398eaeb39165f7e3fa1989bdbb5c1a1d7dd" exitCode=60
Oct 31 20:00:16.968104 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.968065    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" event={"ID":"64add86d-fbd5-4293-8600-34d61126b4ec","Type":"ContainerDied","Data":"85123cab1866c7d571f7c6767920d398eaeb39165f7e3fa1989bdbb5c1a1d7dd"}
Oct 31 20:00:16.968482 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.968462    5006 scope.go:122] "RemoveContainer" containerID="85123cab1866c7d571f7c6767920d398eaeb39165f7e3fa1989bdbb5c1a1d7dd"
Oct 31 20:00:16.968845 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:16.968824    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q_odh-ai-gateway-infra(64add86d-fbd5-4293-8600-34d61126b4ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" podUID="64add86d-fbd5-4293-8600-34d61126b4ec"
Oct 31 20:00:16.970232 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.970207    5006 generic.go:356] "Generic (PLEG): container finished" podID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerID="087d870c015f79cf2226daaae3804a22262c79939f4c3ed346687aa2a3207f7b" exitCode=60
Oct 31 20:00:16.970327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.970267    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" event={"ID":"a47903d5-953b-44c2-9b07-a34c283a21b4","Type":"ContainerDied","Data":"087d870c015f79cf2226daaae3804a22262c79939f4c3ed346687aa2a3207f7b"}
Oct 31 20:00:16.970638 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.970625    5006 scope.go:122] "RemoveContainer" containerID="087d870c015f79cf2226daaae3804a22262c79939f4c3ed346687aa2a3207f7b"
Oct 31 20:00:16.970958 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:16.970941    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj_odh-ai-gateway-infra(a47903d5-953b-44c2-9b07-a34c283a21b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4"
Oct 31 20:00:16.979769 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.979733    5006 scope.go:122] "RemoveContainer" containerID="072923a4d8603d895cb76e9675ecda7683505668c1a9275a9a52219362ea839f"
Oct 31 20:00:16.995978 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:16.995953    5006 scope.go:122] "RemoveContainer" containerID="ca38786461440ae02a77addf198f4d2115916bbd4bc5c8690a91c6cac18da066"
Oct 31 20:00:17.591517 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.591481    5006 scope.go:122] "RemoveContainer" containerID="89df1042a616063e88f9b24bf087ed56a2b00d2f6f464ca697feda0088a247a5"
Oct 31 20:00:17.591749 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.591620    5006 scope.go:122] "RemoveContainer" containerID="ab97153a24b4e2b6c9a9084a2cfef1e486ca04c4ccc39802dce5c19fe0cf9f2a"
Oct 31 20:00:17.591908 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.591890    5006 scope.go:122] "RemoveContainer" containerID="562b2921b4e7555be7cba39215f550e0c73299938b157d2997242b9d861e98a3"
Oct 31 20:00:17.591989 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.591976    5006 scope.go:122] "RemoveContainer" containerID="6ef0773881e5dda0a0855168fa4528b73cb786120f78d2c275ab837d892e5332"
Oct 31 20:00:17.592432 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.592301    5006 scope.go:122] "RemoveContainer" containerID="c092ac495ee1cc4e9bc1f32e0537c45305d74340535e309d30ace1b2e4a9084d"
Oct 31 20:00:17.983005 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.982961    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" event={"ID":"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca","Type":"ContainerStarted","Data":"49289f6cfdb6a3f87f62d81744147268d95f11c139db70ad40b99b63679cef2b"}
Oct 31 20:00:17.984910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.984883    5006 generic.go:356] "Generic (PLEG): container finished" podID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerID="36484c4d872723bba78afefc61eccf1e8622cf7e18dd36e23020803a1bd86e89" exitCode=60
Oct 31 20:00:17.985071 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.984953    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" event={"ID":"4cc97d1e-ef12-49d3-839c-cb2fb8766213","Type":"ContainerDied","Data":"36484c4d872723bba78afefc61eccf1e8622cf7e18dd36e23020803a1bd86e89"}
Oct 31 20:00:17.985071 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.985001    5006 scope.go:122] "RemoveContainer" containerID="25b8f401cfdf3bed07048e27c937fd9edbd6ede46a91524ac1591712fc847be3"
Oct 31 20:00:17.985411 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.985396    5006 scope.go:122] "RemoveContainer" containerID="36484c4d872723bba78afefc61eccf1e8622cf7e18dd36e23020803a1bd86e89"
Oct 31 20:00:17.985820 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:17.985799    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv_odh-ai-gateway-infra(4cc97d1e-ef12-49d3-839c-cb2fb8766213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213"
Oct 31 20:00:17.987039 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.987018    5006 generic.go:356] "Generic (PLEG): container finished" podID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerID="cc9ca7b41e20ed2768f10dd6f869dc5802a7259a2ff57b0315048107279ec614" exitCode=60
Oct 31 20:00:17.987118 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.987039    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5" event={"ID":"81f82c2d-61ce-409c-8fb3-dadb3e0efb43","Type":"ContainerDied","Data":"cc9ca7b41e20ed2768f10dd6f869dc5802a7259a2ff57b0315048107279ec614"}
Oct 31 20:00:17.987417 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:17.987395    5006 scope.go:122] "RemoveContainer" containerID="cc9ca7b41e20ed2768f10dd6f869dc5802a7259a2ff57b0315048107279ec614"
Oct 31 20:00:17.987748 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:17.987728    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5_odh-ai-gateway-infra(81f82c2d-61ce-409c-8fb3-dadb3e0efb43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43"
Oct 31 20:00:18.092256 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.092228    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"]
Oct 31 20:00:18.208214 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.208177    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"]
Oct 31 20:00:18.212601 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.212576    5006 scope.go:122] "RemoveContainer" containerID="328ad9ddec90454827a632b4ac6fe8a79f49204e00a00af9a19ab5dfe2573410"
Oct 31 20:00:18.290448 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.290352    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"
Oct 31 20:00:18.330070 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.330040    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"]
Oct 31 20:00:18.352111 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.352078    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-kube-api-access-jx5lm\" (UniqueName: \"kubernetes.io/projected/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-kube-api-access-jx5lm\") pod \"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae\" (UID: \"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae\") "
Oct 31 20:00:18.352838 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.352810    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-maas-api-service-ca\") pod \"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae\" (UID: \"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae\") "
Oct 31 20:00:18.353434 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.353406    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-maas-api-service-ca" pod "dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" (UID: "dda3dee4-ae07-418b-b5e3-a1dd2fb312ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:18.353859 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.353838    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:18.356970 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.356940    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-kube-api-access-jx5lm" pod "dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" (UID: "dda3dee4-ae07-418b-b5e3-a1dd2fb312ae"). InnerVolumeSpecName "kube-api-access-jx5lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:18.375700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.374402    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"
Oct 31 20:00:18.454954 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.454861    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64add86d-fbd5-4293-8600-34d61126b4ec-kube-api-access-lkntw\" (UniqueName: \"kubernetes.io/projected/64add86d-fbd5-4293-8600-34d61126b4ec-kube-api-access-lkntw\") pod \"64add86d-fbd5-4293-8600-34d61126b4ec\" (UID: \"64add86d-fbd5-4293-8600-34d61126b4ec\") "
Oct 31 20:00:18.455124 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.455080    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64add86d-fbd5-4293-8600-34d61126b4ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64add86d-fbd5-4293-8600-34d61126b4ec-maas-api-service-ca\") pod \"64add86d-fbd5-4293-8600-34d61126b4ec\" (UID: \"64add86d-fbd5-4293-8600-34d61126b4ec\") "
Oct 31 20:00:18.455673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.455513    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64add86d-fbd5-4293-8600-34d61126b4ec-maas-api-service-ca" pod "64add86d-fbd5-4293-8600-34d61126b4ec" (UID: "64add86d-fbd5-4293-8600-34d61126b4ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:18.456012 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.455982    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64add86d-fbd5-4293-8600-34d61126b4ec-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:18.456125 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.456017    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx5lm\" (UniqueName: \"kubernetes.io/projected/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae-kube-api-access-jx5lm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:18.457662 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.457620    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64add86d-fbd5-4293-8600-34d61126b4ec-kube-api-access-lkntw" pod "64add86d-fbd5-4293-8600-34d61126b4ec" (UID: "64add86d-fbd5-4293-8600-34d61126b4ec"). InnerVolumeSpecName "kube-api-access-lkntw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:18.493231 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.493203    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"
Oct 31 20:00:18.557287 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.557251    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47903d5-953b-44c2-9b07-a34c283a21b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47903d5-953b-44c2-9b07-a34c283a21b4-maas-api-service-ca\") pod \"a47903d5-953b-44c2-9b07-a34c283a21b4\" (UID: \"a47903d5-953b-44c2-9b07-a34c283a21b4\") "
Oct 31 20:00:18.557466 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.557449    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47903d5-953b-44c2-9b07-a34c283a21b4-kube-api-access-hdc5l\" (UniqueName: \"kubernetes.io/projected/a47903d5-953b-44c2-9b07-a34c283a21b4-kube-api-access-hdc5l\") pod \"a47903d5-953b-44c2-9b07-a34c283a21b4\" (UID: \"a47903d5-953b-44c2-9b07-a34c283a21b4\") "
Oct 31 20:00:18.557732 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.557707    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47903d5-953b-44c2-9b07-a34c283a21b4-maas-api-service-ca" pod "a47903d5-953b-44c2-9b07-a34c283a21b4" (UID: "a47903d5-953b-44c2-9b07-a34c283a21b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:18.557915 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.557899    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47903d5-953b-44c2-9b07-a34c283a21b4-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:18.557969 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.557922    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkntw\" (UniqueName: \"kubernetes.io/projected/64add86d-fbd5-4293-8600-34d61126b4ec-kube-api-access-lkntw\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:18.559568 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.559549    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47903d5-953b-44c2-9b07-a34c283a21b4-kube-api-access-hdc5l" pod "a47903d5-953b-44c2-9b07-a34c283a21b4" (UID: "a47903d5-953b-44c2-9b07-a34c283a21b4"). InnerVolumeSpecName "kube-api-access-hdc5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:18.591544 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.591510    5006 scope.go:122] "RemoveContainer" containerID="4da96495800f526f0e37d7e6ac590147e862fc34c16f831e42bd2ce0e8b860c3"
Oct 31 20:00:18.659830 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:18.659798    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdc5l\" (UniqueName: \"kubernetes.io/projected/a47903d5-953b-44c2-9b07-a34c283a21b4-kube-api-access-hdc5l\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:19.012773 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.012737    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"
Oct 31 20:00:19.012773 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.012751    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q" event={"ID":"64add86d-fbd5-4293-8600-34d61126b4ec","Type":"ContainerDied","Data":"9b0bd9d6b83c910958f5baedfd6d566abc0b399b4325f61bedf617f7152cf65e"}
Oct 31 20:00:19.013310 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.012827    5006 scope.go:122] "RemoveContainer" containerID="85123cab1866c7d571f7c6767920d398eaeb39165f7e3fa1989bdbb5c1a1d7dd"
Oct 31 20:00:19.016021 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.015993    5006 generic.go:356] "Generic (PLEG): container finished" podID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerID="5c10b1400c46efdd025faf41a8213d77988f6ab58c3f4c6116ffd40588c58833" exitCode=60
Oct 31 20:00:19.016161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.016072    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" event={"ID":"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8","Type":"ContainerDied","Data":"5c10b1400c46efdd025faf41a8213d77988f6ab58c3f4c6116ffd40588c58833"}
Oct 31 20:00:19.016633 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.016611    5006 scope.go:122] "RemoveContainer" containerID="5c10b1400c46efdd025faf41a8213d77988f6ab58c3f4c6116ffd40588c58833"
Oct 31 20:00:19.017074 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:19.017021    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq_odh-ai-gateway-infra(0e171a20-81b9-4f9b-8b74-96b9b05bf4a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8"
Oct 31 20:00:19.018574 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.018549    5006 generic.go:356] "Generic (PLEG): container finished" podID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerID="49289f6cfdb6a3f87f62d81744147268d95f11c139db70ad40b99b63679cef2b" exitCode=60
Oct 31 20:00:19.018691 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.018623    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" event={"ID":"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca","Type":"ContainerDied","Data":"49289f6cfdb6a3f87f62d81744147268d95f11c139db70ad40b99b63679cef2b"}
Oct 31 20:00:19.019158 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.019137    5006 scope.go:122] "RemoveContainer" containerID="49289f6cfdb6a3f87f62d81744147268d95f11c139db70ad40b99b63679cef2b"
Oct 31 20:00:19.019520 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:19.019492    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l_odh-ai-gateway-infra(99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca"
Oct 31 20:00:19.026144 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.026105    5006 generic.go:356] "Generic (PLEG): container finished" podID="339c95d8-6136-4386-87aa-6da89376a3a2" containerID="9e19e2d64199ba96b1ba216f6b74ddc069301e27b2b92c32ef7652344d5329e4" exitCode=60
Oct 31 20:00:19.026247 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.026213    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" event={"ID":"339c95d8-6136-4386-87aa-6da89376a3a2","Type":"ContainerDied","Data":"9e19e2d64199ba96b1ba216f6b74ddc069301e27b2b92c32ef7652344d5329e4"}
Oct 31 20:00:19.026628 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.026611    5006 scope.go:122] "RemoveContainer" containerID="9e19e2d64199ba96b1ba216f6b74ddc069301e27b2b92c32ef7652344d5329e4"
Oct 31 20:00:19.026985 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:19.026967    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs_odh-ai-gateway-infra(339c95d8-6136-4386-87aa-6da89376a3a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" podUID="339c95d8-6136-4386-87aa-6da89376a3a2"
Oct 31 20:00:19.027342 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.027308    5006 scope.go:122] "RemoveContainer" containerID="562b2921b4e7555be7cba39215f550e0c73299938b157d2997242b9d861e98a3"
Oct 31 20:00:19.029247 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.029223    5006 generic.go:356] "Generic (PLEG): container finished" podID="3d56232f-527a-4659-9756-99c052e97f1f" containerID="a1cc441841c8c7355f91155ea74701483ab766e570971c33477c5c68a2c577c0" exitCode=60
Oct 31 20:00:19.029337 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.029285    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" event={"ID":"3d56232f-527a-4659-9756-99c052e97f1f","Type":"ContainerDied","Data":"a1cc441841c8c7355f91155ea74701483ab766e570971c33477c5c68a2c577c0"}
Oct 31 20:00:19.029738 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.029719    5006 scope.go:122] "RemoveContainer" containerID="a1cc441841c8c7355f91155ea74701483ab766e570971c33477c5c68a2c577c0"
Oct 31 20:00:19.030052 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:19.030027    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c_odh-ai-gateway-infra(3d56232f-527a-4659-9756-99c052e97f1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" podUID="3d56232f-527a-4659-9756-99c052e97f1f"
Oct 31 20:00:19.031261 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.031241    5006 generic.go:356] "Generic (PLEG): container finished" podID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerID="eea64e99908a8ea75c81116d28c64c88ad07b7f6e5c927ef605ae560b4144f1f" exitCode=60
Oct 31 20:00:19.031360 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.031317    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" event={"ID":"8beb49e6-eb6d-4398-a6a2-f32f176165a1","Type":"ContainerDied","Data":"eea64e99908a8ea75c81116d28c64c88ad07b7f6e5c927ef605ae560b4144f1f"}
Oct 31 20:00:19.031701 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.031685    5006 scope.go:122] "RemoveContainer" containerID="eea64e99908a8ea75c81116d28c64c88ad07b7f6e5c927ef605ae560b4144f1f"
Oct 31 20:00:19.031986 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:19.031965    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4_odh-ai-gateway-infra(8beb49e6-eb6d-4398-a6a2-f32f176165a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1"
Oct 31 20:00:19.032840 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.032822    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"
Oct 31 20:00:19.032974 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.032867    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj" event={"ID":"a47903d5-953b-44c2-9b07-a34c283a21b4","Type":"ContainerDied","Data":"aa07095f38cf9885d9f1cd9b6f8344176fb07152ee1a7ce367a1cbc19d9a2dc9"}
Oct 31 20:00:19.034347 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.034331    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"
Oct 31 20:00:19.034438 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.034350    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4" event={"ID":"dda3dee4-ae07-418b-b5e3-a1dd2fb312ae","Type":"ContainerDied","Data":"d76ceac1cea21edf41d1df358a70c3b261fdb53fb072c8764a8033b3a51485ee"}
Oct 31 20:00:19.039025 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.038991    5006 generic.go:356] "Generic (PLEG): container finished" podID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerID="39aae86236587cfb9cc58d670ab1caf7b49cfd53f5ec82779db16e0a272fd53b" exitCode=60
Oct 31 20:00:19.039182 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.039054    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" event={"ID":"1cea76a4-21e8-44b7-b9df-b7f19667afb9","Type":"ContainerDied","Data":"39aae86236587cfb9cc58d670ab1caf7b49cfd53f5ec82779db16e0a272fd53b"}
Oct 31 20:00:19.039609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.039584    5006 scope.go:122] "RemoveContainer" containerID="39aae86236587cfb9cc58d670ab1caf7b49cfd53f5ec82779db16e0a272fd53b"
Oct 31 20:00:19.040035 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:19.040009    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc_odh-ai-gateway-infra(1cea76a4-21e8-44b7-b9df-b7f19667afb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9"
Oct 31 20:00:19.049421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.049398    5006 scope.go:122] "RemoveContainer" containerID="9ab61bc47bcea0d8c0b1f1ceb36dbf2a9c395abaa18c37e10e98411fecdd1856"
Oct 31 20:00:19.064520 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.064498    5006 scope.go:122] "RemoveContainer" containerID="ab97153a24b4e2b6c9a9084a2cfef1e486ca04c4ccc39802dce5c19fe0cf9f2a"
Oct 31 20:00:19.080978 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.080954    5006 scope.go:122] "RemoveContainer" containerID="c092ac495ee1cc4e9bc1f32e0537c45305d74340535e309d30ace1b2e4a9084d"
Oct 31 20:00:19.095909 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.095884    5006 scope.go:122] "RemoveContainer" containerID="89df1042a616063e88f9b24bf087ed56a2b00d2f6f464ca697feda0088a247a5"
Oct 31 20:00:19.114102 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.114073    5006 scope.go:122] "RemoveContainer" containerID="087d870c015f79cf2226daaae3804a22262c79939f4c3ed346687aa2a3207f7b"
Oct 31 20:00:19.119802 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.119763    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"]
Oct 31 20:00:19.125317 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.125288    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:19.126303 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.126282    5006 scope.go:122] "RemoveContainer" containerID="f94c2301065ba8c82a9dddee6ad2900532ea3673f38b09463437dac317c02448"
Oct 31 20:00:19.137587 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.137547    5006 scope.go:122] "RemoveContainer" containerID="6ef0773881e5dda0a0855168fa4528b73cb786120f78d2c275ab837d892e5332"
Oct 31 20:00:19.397673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.393744    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"]
Oct 31 20:00:19.487063 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.487024    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"]
Oct 31 20:00:19.590946 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.590912    5006 scope.go:122] "RemoveContainer" containerID="f78068e31be99db856262448654d59ed02bd78202c80252d56434df14d4c8edf"
Oct 31 20:00:19.659502 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.659467    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"
Oct 31 20:00:19.814642 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.814607    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc97d1e-ef12-49d3-839c-cb2fb8766213-kube-api-access-s2r8p\" (UniqueName: \"kubernetes.io/projected/4cc97d1e-ef12-49d3-839c-cb2fb8766213-kube-api-access-s2r8p\") pod \"4cc97d1e-ef12-49d3-839c-cb2fb8766213\" (UID: \"4cc97d1e-ef12-49d3-839c-cb2fb8766213\") "
Oct 31 20:00:19.815183 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.815154    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc97d1e-ef12-49d3-839c-cb2fb8766213-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc97d1e-ef12-49d3-839c-cb2fb8766213-maas-api-service-ca\") pod \"4cc97d1e-ef12-49d3-839c-cb2fb8766213\" (UID: \"4cc97d1e-ef12-49d3-839c-cb2fb8766213\") "
Oct 31 20:00:19.816308 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.816284    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc97d1e-ef12-49d3-839c-cb2fb8766213-maas-api-service-ca" pod "4cc97d1e-ef12-49d3-839c-cb2fb8766213" (UID: "4cc97d1e-ef12-49d3-839c-cb2fb8766213"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:19.818452 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.818410    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc97d1e-ef12-49d3-839c-cb2fb8766213-kube-api-access-s2r8p" pod "4cc97d1e-ef12-49d3-839c-cb2fb8766213" (UID: "4cc97d1e-ef12-49d3-839c-cb2fb8766213"). InnerVolumeSpecName "kube-api-access-s2r8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:19.822760 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.822731    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"]
Oct 31 20:00:19.823670 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.823635    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-662be1-29891280-rm6d4"]
Oct 31 20:00:19.859905 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.859879    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"
Oct 31 20:00:19.917246 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.917160    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc97d1e-ef12-49d3-839c-cb2fb8766213-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:19.917246 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.917191    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2r8p\" (UniqueName: \"kubernetes.io/projected/4cc97d1e-ef12-49d3-839c-cb2fb8766213-kube-api-access-s2r8p\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:19.929475 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.929441    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"]
Oct 31 20:00:19.933144 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:19.933119    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ddb5a8-29891280-c4jqj"]
Oct 31 20:00:20.018208 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.018173    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-maas-api-service-ca\") pod \"81f82c2d-61ce-409c-8fb3-dadb3e0efb43\" (UID: \"81f82c2d-61ce-409c-8fb3-dadb3e0efb43\") "
Oct 31 20:00:20.018758 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.018309    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-kube-api-access-q4kw5\" (UniqueName: \"kubernetes.io/projected/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-kube-api-access-q4kw5\") pod \"81f82c2d-61ce-409c-8fb3-dadb3e0efb43\" (UID: \"81f82c2d-61ce-409c-8fb3-dadb3e0efb43\") "
Oct 31 20:00:20.018758 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.018575    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-maas-api-service-ca" pod "81f82c2d-61ce-409c-8fb3-dadb3e0efb43" (UID: "81f82c2d-61ce-409c-8fb3-dadb3e0efb43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:20.019113 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.019070    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.021887 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.021858    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-kube-api-access-q4kw5" pod "81f82c2d-61ce-409c-8fb3-dadb3e0efb43" (UID: "81f82c2d-61ce-409c-8fb3-dadb3e0efb43"). InnerVolumeSpecName "kube-api-access-q4kw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:20.043089 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.043050    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"]
Oct 31 20:00:20.048481 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.048134    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902ffd9-29891280-l574q"]
Oct 31 20:00:20.059475 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.059414    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" event={"ID":"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a","Type":"ContainerStarted","Data":"fbb09f39c30026bc633a9c1937f3ab9824347a510e3d1eb9a9ae0a647d073292"}
Oct 31 20:00:20.066384 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.066357    5006 generic.go:356] "Generic (PLEG): container finished" podID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerID="5373de5f021cc038a1d91fc602a19538906a8ce578b1531c2c5b7acfd8521971" exitCode=60
Oct 31 20:00:20.066524 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.066428    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" event={"ID":"43664432-b3d5-46ad-9032-bfd6b5b0d584","Type":"ContainerDied","Data":"5373de5f021cc038a1d91fc602a19538906a8ce578b1531c2c5b7acfd8521971"}
Oct 31 20:00:20.066524 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.066481    5006 scope.go:122] "RemoveContainer" containerID="4da96495800f526f0e37d7e6ac590147e862fc34c16f831e42bd2ce0e8b860c3"
Oct 31 20:00:20.066965 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.066938    5006 scope.go:122] "RemoveContainer" containerID="5373de5f021cc038a1d91fc602a19538906a8ce578b1531c2c5b7acfd8521971"
Oct 31 20:00:20.067316 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:20.067291    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5_odh-ai-gateway-infra(43664432-b3d5-46ad-9032-bfd6b5b0d584)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584"
Oct 31 20:00:20.070957 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.070940    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"
Oct 31 20:00:20.071068 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.070958    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv" event={"ID":"4cc97d1e-ef12-49d3-839c-cb2fb8766213","Type":"ContainerDied","Data":"65127f42609aa85508b7a04a825509c34ea7437689f5fa9274047f4a6d89121d"}
Oct 31 20:00:20.072618 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.072596    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"
Oct 31 20:00:20.072751 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.072593    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5" event={"ID":"81f82c2d-61ce-409c-8fb3-dadb3e0efb43","Type":"ContainerDied","Data":"2980b2a1f7d10d83cc9847eea6fab58e2ad62d21c5bac4ca23daa4a8afa50da8"}
Oct 31 20:00:20.083080 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.083059    5006 scope.go:122] "RemoveContainer" containerID="36484c4d872723bba78afefc61eccf1e8622cf7e18dd36e23020803a1bd86e89"
Oct 31 20:00:20.096339 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.096312    5006 scope.go:122] "RemoveContainer" containerID="cc9ca7b41e20ed2768f10dd6f869dc5802a7259a2ff57b0315048107279ec614"
Oct 31 20:00:20.120311 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.120267    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4kw5\" (UniqueName: \"kubernetes.io/projected/81f82c2d-61ce-409c-8fb3-dadb3e0efb43-kube-api-access-q4kw5\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.198158 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.198041    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"]
Oct 31 20:00:20.264917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.264893    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"
Oct 31 20:00:20.331824 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.328749    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"]
Oct 31 20:00:20.333752 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.333722    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"]
Oct 31 20:00:20.337687 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.337660    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9a8346-29891280-nfbmv"]
Oct 31 20:00:20.373773 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.373745    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"
Oct 31 20:00:20.424763 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.424723    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-kube-api-access-xf2lf\" (UniqueName: \"kubernetes.io/projected/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-kube-api-access-xf2lf\") pod \"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca\" (UID: \"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca\") "
Oct 31 20:00:20.424924 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.424794    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-maas-api-service-ca\") pod \"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca\" (UID: \"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca\") "
Oct 31 20:00:20.425850 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.425819    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-maas-api-service-ca" pod "99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" (UID: "99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:20.428357 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.428334    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-kube-api-access-xf2lf" pod "99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" (UID: "99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca"). InnerVolumeSpecName "kube-api-access-xf2lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:20.461367 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.461126    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"]
Oct 31 20:00:20.496233 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.496198    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"
Oct 31 20:00:20.516238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.516164    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"]
Oct 31 20:00:20.519736 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.519705    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-570915-29891280-pctm5"]
Oct 31 20:00:20.526716 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.526628    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d56232f-527a-4659-9756-99c052e97f1f-kube-api-access-2l26f\" (UniqueName: \"kubernetes.io/projected/3d56232f-527a-4659-9756-99c052e97f1f-kube-api-access-2l26f\") pod \"3d56232f-527a-4659-9756-99c052e97f1f\" (UID: \"3d56232f-527a-4659-9756-99c052e97f1f\") "
Oct 31 20:00:20.526877 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.526821    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d56232f-527a-4659-9756-99c052e97f1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d56232f-527a-4659-9756-99c052e97f1f-maas-api-service-ca\") pod \"3d56232f-527a-4659-9756-99c052e97f1f\" (UID: \"3d56232f-527a-4659-9756-99c052e97f1f\") "
Oct 31 20:00:20.527350 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.527255    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf2lf\" (UniqueName: \"kubernetes.io/projected/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-kube-api-access-xf2lf\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.527350 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.527270    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.527350 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.527315    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d56232f-527a-4659-9756-99c052e97f1f-maas-api-service-ca" pod "3d56232f-527a-4659-9756-99c052e97f1f" (UID: "3d56232f-527a-4659-9756-99c052e97f1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:20.529159 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.529137    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d56232f-527a-4659-9756-99c052e97f1f-kube-api-access-2l26f" pod "3d56232f-527a-4659-9756-99c052e97f1f" (UID: "3d56232f-527a-4659-9756-99c052e97f1f"). InnerVolumeSpecName "kube-api-access-2l26f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:20.586508 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.586467    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"]
Oct 31 20:00:20.617054 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.617020    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"
Oct 31 20:00:20.628185 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.628148    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cea76a4-21e8-44b7-b9df-b7f19667afb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cea76a4-21e8-44b7-b9df-b7f19667afb9-maas-api-service-ca\") pod \"1cea76a4-21e8-44b7-b9df-b7f19667afb9\" (UID: \"1cea76a4-21e8-44b7-b9df-b7f19667afb9\") "
Oct 31 20:00:20.628506 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.628485    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cea76a4-21e8-44b7-b9df-b7f19667afb9-kube-api-access-4r27t\" (UniqueName: \"kubernetes.io/projected/1cea76a4-21e8-44b7-b9df-b7f19667afb9-kube-api-access-4r27t\") pod \"1cea76a4-21e8-44b7-b9df-b7f19667afb9\" (UID: \"1cea76a4-21e8-44b7-b9df-b7f19667afb9\") "
Oct 31 20:00:20.628666 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.628491    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cea76a4-21e8-44b7-b9df-b7f19667afb9-maas-api-service-ca" pod "1cea76a4-21e8-44b7-b9df-b7f19667afb9" (UID: "1cea76a4-21e8-44b7-b9df-b7f19667afb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:20.629113 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.629090    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d56232f-527a-4659-9756-99c052e97f1f-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.629213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.629118    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cea76a4-21e8-44b7-b9df-b7f19667afb9-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.629213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.629133    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l26f\" (UniqueName: \"kubernetes.io/projected/3d56232f-527a-4659-9756-99c052e97f1f-kube-api-access-2l26f\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.630803 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.630774    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cea76a4-21e8-44b7-b9df-b7f19667afb9-kube-api-access-4r27t" pod "1cea76a4-21e8-44b7-b9df-b7f19667afb9" (UID: "1cea76a4-21e8-44b7-b9df-b7f19667afb9"). InnerVolumeSpecName "kube-api-access-4r27t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:20.705038 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.704203    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"]
Oct 31 20:00:20.729859 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.729770    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339c95d8-6136-4386-87aa-6da89376a3a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339c95d8-6136-4386-87aa-6da89376a3a2-maas-api-service-ca\") pod \"339c95d8-6136-4386-87aa-6da89376a3a2\" (UID: \"339c95d8-6136-4386-87aa-6da89376a3a2\") "
Oct 31 20:00:20.730004 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.729945    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339c95d8-6136-4386-87aa-6da89376a3a2-kube-api-access-bk74w\" (UniqueName: \"kubernetes.io/projected/339c95d8-6136-4386-87aa-6da89376a3a2-kube-api-access-bk74w\") pod \"339c95d8-6136-4386-87aa-6da89376a3a2\" (UID: \"339c95d8-6136-4386-87aa-6da89376a3a2\") "
Oct 31 20:00:20.730410 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.730383    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339c95d8-6136-4386-87aa-6da89376a3a2-maas-api-service-ca" pod "339c95d8-6136-4386-87aa-6da89376a3a2" (UID: "339c95d8-6136-4386-87aa-6da89376a3a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:20.730568 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.730485    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r27t\" (UniqueName: \"kubernetes.io/projected/1cea76a4-21e8-44b7-b9df-b7f19667afb9-kube-api-access-4r27t\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.730568 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.730503    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339c95d8-6136-4386-87aa-6da89376a3a2-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.732411 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.732380    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339c95d8-6136-4386-87aa-6da89376a3a2-kube-api-access-bk74w" pod "339c95d8-6136-4386-87aa-6da89376a3a2" (UID: "339c95d8-6136-4386-87aa-6da89376a3a2"). InnerVolumeSpecName "kube-api-access-bk74w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:20.756736 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.756710    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"
Oct 31 20:00:20.834698 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.831909    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk74w\" (UniqueName: \"kubernetes.io/projected/339c95d8-6136-4386-87aa-6da89376a3a2-kube-api-access-bk74w\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.859492 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.859379    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:20.876001 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.875973    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"
Oct 31 20:00:20.932519 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.932488    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-kube-api-access-rnq5j\" (UniqueName: \"kubernetes.io/projected/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-kube-api-access-rnq5j\") pod \"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8\" (UID: \"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8\") "
Oct 31 20:00:20.932696 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.932547    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-maas-api-service-ca\") pod \"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8\" (UID: \"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8\") "
Oct 31 20:00:20.932963 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.932938    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-maas-api-service-ca" pod "0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" (UID: "0e171a20-81b9-4f9b-8b74-96b9b05bf4a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:20.934763 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:20.934741    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-kube-api-access-rnq5j" pod "0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" (UID: "0e171a20-81b9-4f9b-8b74-96b9b05bf4a8"). InnerVolumeSpecName "kube-api-access-rnq5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:21.033848 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.033764    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8beb49e6-eb6d-4398-a6a2-f32f176165a1-kube-api-access-jzjnt\" (UniqueName: \"kubernetes.io/projected/8beb49e6-eb6d-4398-a6a2-f32f176165a1-kube-api-access-jzjnt\") pod \"8beb49e6-eb6d-4398-a6a2-f32f176165a1\" (UID: \"8beb49e6-eb6d-4398-a6a2-f32f176165a1\") "
Oct 31 20:00:21.034304 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.033925    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8beb49e6-eb6d-4398-a6a2-f32f176165a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8beb49e6-eb6d-4398-a6a2-f32f176165a1-maas-api-service-ca\") pod \"8beb49e6-eb6d-4398-a6a2-f32f176165a1\" (UID: \"8beb49e6-eb6d-4398-a6a2-f32f176165a1\") "
Oct 31 20:00:21.034374 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.034344    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8beb49e6-eb6d-4398-a6a2-f32f176165a1-maas-api-service-ca" pod "8beb49e6-eb6d-4398-a6a2-f32f176165a1" (UID: "8beb49e6-eb6d-4398-a6a2-f32f176165a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:21.034415 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.034386    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:21.034415 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.034400    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnq5j\" (UniqueName: \"kubernetes.io/projected/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8-kube-api-access-rnq5j\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:21.036031 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.036010    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8beb49e6-eb6d-4398-a6a2-f32f176165a1-kube-api-access-jzjnt" pod "8beb49e6-eb6d-4398-a6a2-f32f176165a1" (UID: "8beb49e6-eb6d-4398-a6a2-f32f176165a1"). InnerVolumeSpecName "kube-api-access-jzjnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:21.086916 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.086884    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"
Oct 31 20:00:21.087096 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.086888    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq" event={"ID":"0e171a20-81b9-4f9b-8b74-96b9b05bf4a8","Type":"ContainerDied","Data":"09de4c13cf3e850387a6d7de12870858bcab30a923d16987a3847b53fc8cdde9"}
Oct 31 20:00:21.087096 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.086985    5006 scope.go:122] "RemoveContainer" containerID="5c10b1400c46efdd025faf41a8213d77988f6ab58c3f4c6116ffd40588c58833"
Oct 31 20:00:21.088501 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.088481    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"
Oct 31 20:00:21.088684 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.088538    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l" event={"ID":"99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca","Type":"ContainerDied","Data":"c835e052089e768d0172e13a67b244a09962f1de19b2541f4c66e2624401cd96"}
Oct 31 20:00:21.093422 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.093397    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"
Oct 31 20:00:21.093422 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.093405    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs" event={"ID":"339c95d8-6136-4386-87aa-6da89376a3a2","Type":"ContainerDied","Data":"67724e8a61206811ece953fc601929d061904669b4ffde4cdc3be80a7ced4eab"}
Oct 31 20:00:21.097827 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.097380    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c" event={"ID":"3d56232f-527a-4659-9756-99c052e97f1f","Type":"ContainerDied","Data":"c4b588bd62394dd89d9a1fbc5b11350a857b95f2d0e4cda0b400ad498b12c2ac"}
Oct 31 20:00:21.097827 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.097488    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"
Oct 31 20:00:21.100266 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.100245    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"
Oct 31 20:00:21.100424 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.100265    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4" event={"ID":"8beb49e6-eb6d-4398-a6a2-f32f176165a1","Type":"ContainerDied","Data":"8d5bba186d4d5ed1598a5f7c3c65dbd6db26b073ce82a3a353cc17248b5d39a4"}
Oct 31 20:00:21.101740 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.101717    5006 scope.go:122] "RemoveContainer" containerID="49289f6cfdb6a3f87f62d81744147268d95f11c139db70ad40b99b63679cef2b"
Oct 31 20:00:21.102465 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.102441    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc" event={"ID":"1cea76a4-21e8-44b7-b9df-b7f19667afb9","Type":"ContainerDied","Data":"684aed584b719dfadee3f89b8261155560bbdac6b3a9140cdf90b8ed886a0aa0"}
Oct 31 20:00:21.102606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.102585    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"
Oct 31 20:00:21.106865 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.106835    5006 generic.go:356] "Generic (PLEG): container finished" podID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerID="fbb09f39c30026bc633a9c1937f3ab9824347a510e3d1eb9a9ae0a647d073292" exitCode=60
Oct 31 20:00:21.106995 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.106901    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" event={"ID":"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a","Type":"ContainerDied","Data":"fbb09f39c30026bc633a9c1937f3ab9824347a510e3d1eb9a9ae0a647d073292"}
Oct 31 20:00:21.107606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.107584    5006 scope.go:122] "RemoveContainer" containerID="fbb09f39c30026bc633a9c1937f3ab9824347a510e3d1eb9a9ae0a647d073292"
Oct 31 20:00:21.107977 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:21.107946    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr_odh-ai-gateway-infra(e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a"
Oct 31 20:00:21.116335 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.116313    5006 scope.go:122] "RemoveContainer" containerID="9e19e2d64199ba96b1ba216f6b74ddc069301e27b2b92c32ef7652344d5329e4"
Oct 31 20:00:21.126812 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.126787    5006 scope.go:122] "RemoveContainer" containerID="a1cc441841c8c7355f91155ea74701483ab766e570971c33477c5c68a2c577c0"
Oct 31 20:00:21.135628 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.135600    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjnt\" (UniqueName: \"kubernetes.io/projected/8beb49e6-eb6d-4398-a6a2-f32f176165a1-kube-api-access-jzjnt\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:21.135628 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.135628    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8beb49e6-eb6d-4398-a6a2-f32f176165a1-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:21.136342 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.136329    5006 scope.go:122] "RemoveContainer" containerID="eea64e99908a8ea75c81116d28c64c88ad07b7f6e5c927ef605ae560b4144f1f"
Oct 31 20:00:21.146329 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.146307    5006 scope.go:122] "RemoveContainer" containerID="39aae86236587cfb9cc58d670ab1caf7b49cfd53f5ec82779db16e0a272fd53b"
Oct 31 20:00:21.156525 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.156503    5006 scope.go:122] "RemoveContainer" containerID="f78068e31be99db856262448654d59ed02bd78202c80252d56434df14d4c8edf"
Oct 31 20:00:21.187427 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.187399    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:21.408130 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.408098    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:21.469327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.469285    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"]
Oct 31 20:00:21.475908 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.475871    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9bbcf7-wmh2t"]
Oct 31 20:00:21.602538 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.602500    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" path="/var/lib/kubelet/pods/4cc97d1e-ef12-49d3-839c-cb2fb8766213/volumes"
Oct 31 20:00:21.603033 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.603014    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64add86d-fbd5-4293-8600-34d61126b4ec" path="/var/lib/kubelet/pods/64add86d-fbd5-4293-8600-34d61126b4ec/volumes"
Oct 31 20:00:21.603410 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.603394    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" path="/var/lib/kubelet/pods/81f82c2d-61ce-409c-8fb3-dadb3e0efb43/volumes"
Oct 31 20:00:21.604130 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.604110    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4" path="/var/lib/kubelet/pods/a47903d5-953b-44c2-9b07-a34c283a21b4/volumes"
Oct 31 20:00:21.604507 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.604489    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" path="/var/lib/kubelet/pods/dda3dee4-ae07-418b-b5e3-a1dd2fb312ae/volumes"
Oct 31 20:00:21.604964 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.604948    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69ee09f-4eed-4ad8-874b-a0d992d43bb5" path="/var/lib/kubelet/pods/e69ee09f-4eed-4ad8-874b-a0d992d43bb5/volumes"
Oct 31 20:00:21.683222 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.682609    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"]
Oct 31 20:00:21.694148 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.694105    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-650cf3-29891280-sdr4l"]
Oct 31 20:00:21.755838 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.755802    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"]
Oct 31 20:00:21.800927 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.800881    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"]
Oct 31 20:00:21.807175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.807130    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1de410-29891280-wb67c"]
Oct 31 20:00:21.871993 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.871956    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"]
Oct 31 20:00:21.911471 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.911397    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"]
Oct 31 20:00:21.923883 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.923848    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a5ba62-29891280-6t4dc"]
Oct 31 20:00:21.946767 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.946695    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"
Oct 31 20:00:21.974318 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.974286    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:21.982410 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.982381    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43664432-b3d5-46ad-9032-bfd6b5b0d584-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43664432-b3d5-46ad-9032-bfd6b5b0d584-maas-api-service-ca\") pod \"43664432-b3d5-46ad-9032-bfd6b5b0d584\" (UID: \"43664432-b3d5-46ad-9032-bfd6b5b0d584\") "
Oct 31 20:00:21.982576 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.982543    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43664432-b3d5-46ad-9032-bfd6b5b0d584-kube-api-access-62j9f\" (UniqueName: \"kubernetes.io/projected/43664432-b3d5-46ad-9032-bfd6b5b0d584-kube-api-access-62j9f\") pod \"43664432-b3d5-46ad-9032-bfd6b5b0d584\" (UID: \"43664432-b3d5-46ad-9032-bfd6b5b0d584\") "
Oct 31 20:00:21.982782 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.982758    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43664432-b3d5-46ad-9032-bfd6b5b0d584-maas-api-service-ca" pod "43664432-b3d5-46ad-9032-bfd6b5b0d584" (UID: "43664432-b3d5-46ad-9032-bfd6b5b0d584"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:21.983077 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.983057    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43664432-b3d5-46ad-9032-bfd6b5b0d584-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:21.984696 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:21.984671    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43664432-b3d5-46ad-9032-bfd6b5b0d584-kube-api-access-62j9f" pod "43664432-b3d5-46ad-9032-bfd6b5b0d584" (UID: "43664432-b3d5-46ad-9032-bfd6b5b0d584"). InnerVolumeSpecName "kube-api-access-62j9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:22.093707 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.085126    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62j9f\" (UniqueName: \"kubernetes.io/projected/43664432-b3d5-46ad-9032-bfd6b5b0d584-kube-api-access-62j9f\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.183555 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.183470    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5" event={"ID":"43664432-b3d5-46ad-9032-bfd6b5b0d584","Type":"ContainerDied","Data":"835dc6af1187e78bf8bfeab493464f355916aa1836f2f97fd42c350d23822582"}
Oct 31 20:00:22.183555 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.183547    5006 scope.go:122] "RemoveContainer" containerID="5373de5f021cc038a1d91fc602a19538906a8ce578b1531c2c5b7acfd8521971"
Oct 31 20:00:22.183853 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.183733    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"
Oct 31 20:00:22.201851 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.201770    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"]
Oct 31 20:00:22.211724 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.208033    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-872fee-29891280-kf9rs"]
Oct 31 20:00:22.317700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.317662    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"]
Oct 31 20:00:22.322357 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.322315    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891280-8f7jq"]
Oct 31 20:00:22.358388 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.358357    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"
Oct 31 20:00:22.389099 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.389056    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-maas-api-service-ca\") pod \"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a\" (UID: \"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a\") "
Oct 31 20:00:22.389291 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.389224    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-kube-api-access-j92mw\" (UniqueName: \"kubernetes.io/projected/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-kube-api-access-j92mw\") pod \"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a\" (UID: \"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a\") "
Oct 31 20:00:22.389501 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.389472    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-maas-api-service-ca" pod "e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" (UID: "e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:22.389631 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.389615    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.391479 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.391449    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-kube-api-access-j92mw" pod "e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" (UID: "e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a"). InnerVolumeSpecName "kube-api-access-j92mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:22.491329 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.491234    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j92mw\" (UniqueName: \"kubernetes.io/projected/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a-kube-api-access-j92mw\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.495451 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.495416    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"]
Oct 31 20:00:22.499268 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.499240    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6eeb83-29891280-cbcq4"]
Oct 31 20:00:22.670543 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.670502    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"]
Oct 31 20:00:22.674848 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:22.674819    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfe658-29891280-rfvm5"]
Oct 31 20:00:23.200117 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.200023    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"
Oct 31 20:00:23.200117 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.200055    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr" event={"ID":"e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a","Type":"ContainerDied","Data":"54fc1e4f15b99bfb3908609dc91bf0f52144917629533df801270aa5e22107c7"}
Oct 31 20:00:23.200725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.200125    5006 scope.go:122] "RemoveContainer" containerID="fbb09f39c30026bc633a9c1937f3ab9824347a510e3d1eb9a9ae0a647d073292"
Oct 31 20:00:23.371662 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.371631    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"]
Oct 31 20:00:23.374859 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.374835    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7cfafd-29891280-hqvgr"]
Oct 31 20:00:23.599602 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.599562    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" path="/var/lib/kubelet/pods/0e171a20-81b9-4f9b-8b74-96b9b05bf4a8/volumes"
Oct 31 20:00:23.600361 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.600334    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" path="/var/lib/kubelet/pods/1cea76a4-21e8-44b7-b9df-b7f19667afb9/volumes"
Oct 31 20:00:23.601036 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.601010    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339c95d8-6136-4386-87aa-6da89376a3a2" path="/var/lib/kubelet/pods/339c95d8-6136-4386-87aa-6da89376a3a2/volumes"
Oct 31 20:00:23.602142 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.602114    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d56232f-527a-4659-9756-99c052e97f1f" path="/var/lib/kubelet/pods/3d56232f-527a-4659-9756-99c052e97f1f/volumes"
Oct 31 20:00:23.602831 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.602809    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584" path="/var/lib/kubelet/pods/43664432-b3d5-46ad-9032-bfd6b5b0d584/volumes"
Oct 31 20:00:23.603463 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.603441    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" path="/var/lib/kubelet/pods/8beb49e6-eb6d-4398-a6a2-f32f176165a1/volumes"
Oct 31 20:00:23.604515 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.604480    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" path="/var/lib/kubelet/pods/99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca/volumes"
Oct 31 20:00:23.605178 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.605153    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" path="/var/lib/kubelet/pods/e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a/volumes"
Oct 31 20:00:23.713564 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.713510    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:23.933853 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.933759    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:23.969644 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.969610    5006 scope.go:122] "RemoveContainer" containerID="16ad8abc7b4745e91b5e6f1a8dfc71a4cc9fab3fd4cb1695ab253384d4ddfbc7"
Oct 31 20:00:23.980160 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:23.980136    5006 scope.go:122] "RemoveContainer" containerID="00bebbbdea78ebe0ed1e3d2f18101aec680fb36236566f86a60fd928b795b085"
Oct 31 20:00:26.395474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:26.395439    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:26.608527 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:26.608490    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:28.000050 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:27.999989    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:28.236545 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:28.236483    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:28.455430 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:28.455398    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:30.426055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.426017    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:30.844447 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.844412    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"]
Oct 31 20:00:30.845066 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845046    5006 cpu_manager.go:414] "containerMap: removing container" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerName="cleanup"
Oct 31 20:00:30.845066 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845065    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845078    5006 cpu_manager.go:414] "containerMap: removing container" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845083    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845090    5006 cpu_manager.go:414] "containerMap: removing container" podUID="339c95d8-6136-4386-87aa-6da89376a3a2" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845098    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339c95d8-6136-4386-87aa-6da89376a3a2" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845105    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845110    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845116    5006 cpu_manager.go:414] "containerMap: removing container" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845121    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845128    5006 cpu_manager.go:414] "containerMap: removing container" podUID="35970881-0106-4496-ad86-d908337d2c7c" containerName="revoke-keys"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845134    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35970881-0106-4496-ad86-d908337d2c7c" containerName="revoke-keys"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845145    5006 cpu_manager.go:414] "containerMap: removing container" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845150    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845157    5006 cpu_manager.go:414] "containerMap: removing container" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845161    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845169    5006 cpu_manager.go:414] "containerMap: removing container" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845176    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845185    5006 cpu_manager.go:414] "containerMap: removing container" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845189    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845208    5006 cpu_manager.go:414] "containerMap: removing container" podUID="64add86d-fbd5-4293-8600-34d61126b4ec" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845213    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64add86d-fbd5-4293-8600-34d61126b4ec" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845220    5006 cpu_manager.go:414] "containerMap: removing container" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerName="cleanup"
Oct 31 20:00:30.845218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845224    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845231    5006 cpu_manager.go:414] "containerMap: removing container" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845235    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845247    5006 cpu_manager.go:414] "containerMap: removing container" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845251    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845260    5006 cpu_manager.go:414] "containerMap: removing container" podUID="3d56232f-527a-4659-9756-99c052e97f1f" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845264    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d56232f-527a-4659-9756-99c052e97f1f" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845271    5006 cpu_manager.go:414] "containerMap: removing container" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845275    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845281    5006 cpu_manager.go:414] "containerMap: removing container" podUID="339c95d8-6136-4386-87aa-6da89376a3a2" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845285    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339c95d8-6136-4386-87aa-6da89376a3a2" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845291    5006 cpu_manager.go:414] "containerMap: removing container" podUID="3d56232f-527a-4659-9756-99c052e97f1f" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845295    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d56232f-527a-4659-9756-99c052e97f1f" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845303    5006 cpu_manager.go:414] "containerMap: removing container" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845308    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845316    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845320    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845328    5006 cpu_manager.go:414] "containerMap: removing container" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845333    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845338    5006 cpu_manager.go:414] "containerMap: removing container" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845342    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845349    5006 cpu_manager.go:414] "containerMap: removing container" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845353    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845359    5006 cpu_manager.go:414] "containerMap: removing container" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845364    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845370    5006 cpu_manager.go:414] "containerMap: removing container" podUID="64add86d-fbd5-4293-8600-34d61126b4ec" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845374    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64add86d-fbd5-4293-8600-34d61126b4ec" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845380    5006 cpu_manager.go:414] "containerMap: removing container" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845384    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845391    5006 cpu_manager.go:414] "containerMap: removing container" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845396    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845401    5006 cpu_manager.go:414] "containerMap: removing container" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845404    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845411    5006 cpu_manager.go:414] "containerMap: removing container" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845415    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845423    5006 cpu_manager.go:414] "containerMap: removing container" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845427    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845435    5006 cpu_manager.go:414] "containerMap: removing container" podUID="3d56232f-527a-4659-9756-99c052e97f1f" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845439    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d56232f-527a-4659-9756-99c052e97f1f" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845447    5006 cpu_manager.go:414] "containerMap: removing container" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845451    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845461    5006 cpu_manager.go:414] "containerMap: removing container" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845466    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845475    5006 cpu_manager.go:414] "containerMap: removing container" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845480    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845486    5006 cpu_manager.go:414] "containerMap: removing container" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845490    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845589    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="35970881-0106-4496-ad86-d908337d2c7c" containerName="revoke-keys"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845597    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845603    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845608    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845613    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845620    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="64add86d-fbd5-4293-8600-34d61126b4ec" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845625    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845630    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845637    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="339c95d8-6136-4386-87aa-6da89376a3a2" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845642    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d56232f-527a-4659-9756-99c052e97f1f" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845662    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845670    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="339c95d8-6136-4386-87aa-6da89376a3a2" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845677    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d56232f-527a-4659-9756-99c052e97f1f" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845682    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845689    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845694    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845699    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845705    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845711    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="64add86d-fbd5-4293-8600-34d61126b4ec" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845715    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845720    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845725    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845731    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cea76a4-21e8-44b7-b9df-b7f19667afb9" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845737    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845742    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845748    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845754    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8beb49e6-eb6d-4398-a6a2-f32f176165a1" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845761    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845766    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerName="cleanup"
Oct 31 20:00:30.845871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.845770    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerName="cleanup"
Oct 31 20:00:30.851480 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.851459    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:30.923501 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.923462    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-credential-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:30.923725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.923529    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:30.923725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.923659    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db340a3a-dca2-42cd-84de-01422845fd59-istiod-ca-cert\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:30.923850 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.923723    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-certs\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:30.923850 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.923810    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qttq2\" (UniqueName: \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-kube-api-access-qttq2\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:30.924048 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.924029    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-envoy\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:30.924186 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.924166    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db340a3a-dca2-42cd-84de-01422845fd59-istio-podinfo\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:30.924242 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.924221    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-data\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:30.924297 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.924285    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-istio-token\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:30.945795 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.945761    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"]
Oct 31 20:00:30.961779 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:30.961727    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fd95d7-openshift-default-dockercfg-cvnc8\""
Oct 31 20:00:31.039676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.039224    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-istio-token\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.039676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.039295    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-credential-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.039676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.039358    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.039676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.039396    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db340a3a-dca2-42cd-84de-01422845fd59-istiod-ca-cert\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.039676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.039437    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-certs\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.039676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.039479    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qttq2\" (UniqueName: \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-kube-api-access-qttq2\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.039676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.039611    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-envoy\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.040131 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.039739    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db340a3a-dca2-42cd-84de-01422845fd59-istio-podinfo\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.040131 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.039773    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-data\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.054675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.040219    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-data\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.054675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.041898    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.054675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.042523    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db340a3a-dca2-42cd-84de-01422845fd59-istiod-ca-cert\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.054675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.042786    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-certs\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.079671 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.055411    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:31.079671 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.057432    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-credential-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.131728 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.127376    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-istio-token\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.152672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.133462    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db340a3a-dca2-42cd-84de-01422845fd59-istio-podinfo\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.152672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.143437    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qttq2\" (UniqueName: \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-kube-api-access-qttq2\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.176049 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.176010    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-envoy\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.327037 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.326998    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:31.467000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.466890    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:31.555819 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.555782    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:31.896941 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:00:31.896903    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb340a3a_dca2_42cd_84de_01422845fd59.slice/crio-9c78c3d7269c969904795a5842d944bd1ff17e012fa4b9d88d26bfe99ba68e63 WatchSource:0}: Error finding container 9c78c3d7269c969904795a5842d944bd1ff17e012fa4b9d88d26bfe99ba68e63: Status 404 returned error can't find the container with id 9c78c3d7269c969904795a5842d944bd1ff17e012fa4b9d88d26bfe99ba68e63
Oct 31 20:00:31.899325 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.899295    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"]
Oct 31 20:00:31.900172 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.900138    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:00:31.900269 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.900203    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:00:31.900269 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:31.900231    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:00:32.279335 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:32.279230    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" event={"ID":"db340a3a-dca2-42cd-84de-01422845fd59","Type":"ContainerStarted","Data":"522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc"}
Oct 31 20:00:32.279501 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:32.279337    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" event={"ID":"db340a3a-dca2-42cd-84de-01422845fd59","Type":"ContainerStarted","Data":"9c78c3d7269c969904795a5842d944bd1ff17e012fa4b9d88d26bfe99ba68e63"}
Oct 31 20:00:32.398631 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:32.398509    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" podStartSLOduration=2.398491765 podStartE2EDuration="2.398491765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:32.398182576 +0000 UTC m=+2650.927716647" watchObservedRunningTime="2026-10-31 20:00:32.398491765 +0000 UTC m=+2650.928025820"
Oct 31 20:00:32.467436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:32.467395    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:32.474605 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:32.474569    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:33.286136 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:33.286102    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:33.287330 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:33.287306    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:00:35.355711 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:35.355638    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:35.400293 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:00:35.400257    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:00:37.075976 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.075942    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"]
Oct 31 20:00:37.076566 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076552    5006 cpu_manager.go:414] "containerMap: removing container" podUID="339c95d8-6136-4386-87aa-6da89376a3a2" containerName="cleanup"
Oct 31 20:00:37.076617 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076567    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339c95d8-6136-4386-87aa-6da89376a3a2" containerName="cleanup"
Oct 31 20:00:37.076617 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076585    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerName="cleanup"
Oct 31 20:00:37.076617 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076589    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c56c53-684b-4c9f-b2b4-4bcfe1e7f95a" containerName="cleanup"
Oct 31 20:00:37.076617 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076602    5006 cpu_manager.go:414] "containerMap: removing container" podUID="64add86d-fbd5-4293-8600-34d61126b4ec" containerName="cleanup"
Oct 31 20:00:37.076617 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076608    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64add86d-fbd5-4293-8600-34d61126b4ec" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076639    5006 cpu_manager.go:414] "containerMap: removing container" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076644    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076667    5006 cpu_manager.go:414] "containerMap: removing container" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076672    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076755    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="43664432-b3d5-46ad-9032-bfd6b5b0d584" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076765    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda3dee4-ae07-418b-b5e3-a1dd2fb312ae" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076776    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e171a20-81b9-4f9b-8b74-96b9b05bf4a8" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076782    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47903d5-953b-44c2-9b07-a34c283a21b4" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076788    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc97d1e-ef12-49d3-839c-cb2fb8766213" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076793    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="64add86d-fbd5-4293-8600-34d61126b4ec" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076800    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="99837d4c-a9df-4ac7-b5b8-a84a9c9c1fca" containerName="cleanup"
Oct 31 20:00:37.076808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076808    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="339c95d8-6136-4386-87aa-6da89376a3a2" containerName="cleanup"
Oct 31 20:00:37.077191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076814    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d56232f-527a-4659-9756-99c052e97f1f" containerName="cleanup"
Oct 31 20:00:37.077191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.076820    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f82c2d-61ce-409c-8fb3-dadb3e0efb43" containerName="cleanup"
Oct 31 20:00:37.080060 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.080042    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.133456 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.133420    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"]
Oct 31 20:00:37.167423 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.167388    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e255ed5e-7shsr\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.167619 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.167513    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2g8\" (UniqueName: \"kubernetes.io/projected/64204cf0-9632-49e2-8e2d-03612a4d6116-kube-api-access-zx2g8\") pod \"maas-api-revoke-sub-e255ed5e-7shsr\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.167709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.167619    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e255ed5e-7shsr\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.179983 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.179927    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:00:37.182857 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.182829    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:00:37.182857 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.182832    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:00:37.183042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.182836    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:00:37.266343 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.266305    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:37.268858 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.268833    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e255ed5e-7shsr\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.268987 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.268940    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2g8\" (UniqueName: \"kubernetes.io/projected/64204cf0-9632-49e2-8e2d-03612a4d6116-kube-api-access-zx2g8\") pod \"maas-api-revoke-sub-e255ed5e-7shsr\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.269030 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.268989    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e255ed5e-7shsr\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.269581 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.269561    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e255ed5e-7shsr\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.271623 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.271597    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e255ed5e-7shsr\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.326543 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.326462    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2g8\" (UniqueName: \"kubernetes.io/projected/64204cf0-9632-49e2-8e2d-03612a4d6116-kube-api-access-zx2g8\") pod \"maas-api-revoke-sub-e255ed5e-7shsr\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.396091 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.396053    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:37.803623 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:37.803591    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"]
Oct 31 20:00:38.009626 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:38.009565    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:38.324923 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:38.324885    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr" event={"ID":"64204cf0-9632-49e2-8e2d-03612a4d6116","Type":"ContainerStarted","Data":"7bc6b2c24c3887fcf84b253e275e3ef9c6774e9aceadd220676671f950db148d"}
Oct 31 20:00:38.325326 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:38.324983    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr" event={"ID":"64204cf0-9632-49e2-8e2d-03612a4d6116","Type":"ContainerStarted","Data":"502c044dd666cf88f10f34c912a3aaeedebc31fb6af1769364ef7c9b3f4c1bca"}
Oct 31 20:00:38.442306 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:38.442246    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr" podStartSLOduration=1.442227179 podStartE2EDuration="1.442227179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:38.439361176 +0000 UTC m=+2656.968895229" watchObservedRunningTime="2026-10-31 20:00:38.442227179 +0000 UTC m=+2656.971761231"
Oct 31 20:00:38.486675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:38.486557    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:38.714868 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:38.714826    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:38.825139 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:38.825104    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"]
Oct 31 20:00:38.828596 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:38.828565    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7c0d34-2w82x"]
Oct 31 20:00:39.304517 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.304476    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:39.334497 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.334464    5006 generic.go:356] "Generic (PLEG): container finished" podID="64204cf0-9632-49e2-8e2d-03612a4d6116" containerID="7bc6b2c24c3887fcf84b253e275e3ef9c6774e9aceadd220676671f950db148d" exitCode=0
Oct 31 20:00:39.334964 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.334551    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr" event={"ID":"64204cf0-9632-49e2-8e2d-03612a4d6116","Type":"ContainerDied","Data":"7bc6b2c24c3887fcf84b253e275e3ef9c6774e9aceadd220676671f950db148d"}
Oct 31 20:00:39.519220 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.519181    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:39.615673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.615464    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652397b0-4c7d-49fe-aab5-fc67a4a9046f" path="/var/lib/kubelet/pods/652397b0-4c7d-49fe-aab5-fc67a4a9046f/volumes"
Oct 31 20:00:39.709605 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.709564    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"]
Oct 31 20:00:39.714882 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.714849    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:00:39.741401 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.741360    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:39.764996 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.764961    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"]
Oct 31 20:00:39.769897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.769861    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fd95d7\""
Oct 31 20:00:39.771921 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.771889    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fd95d7-dockercfg-srkgq\""
Oct 31 20:00:39.846716 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.816584    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7330358-d4c3-4f17-a61c-d794d4425a28-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj\" (UID: \"e7330358-d4c3-4f17-a61c-d794d4425a28\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:00:39.846716 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.816663    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbd28\" (UniqueName: \"kubernetes.io/projected/e7330358-d4c3-4f17-a61c-d794d4425a28-kube-api-access-rbd28\") pod \"payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj\" (UID: \"e7330358-d4c3-4f17-a61c-d794d4425a28\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:00:39.918715 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.918558    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7330358-d4c3-4f17-a61c-d794d4425a28-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj\" (UID: \"e7330358-d4c3-4f17-a61c-d794d4425a28\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:00:39.918715 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.918620    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbd28\" (UniqueName: \"kubernetes.io/projected/e7330358-d4c3-4f17-a61c-d794d4425a28-kube-api-access-rbd28\") pod \"payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj\" (UID: \"e7330358-d4c3-4f17-a61c-d794d4425a28\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:00:39.919968 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.919940    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7330358-d4c3-4f17-a61c-d794d4425a28-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj\" (UID: \"e7330358-d4c3-4f17-a61c-d794d4425a28\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:00:39.957758 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.957725    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"]
Oct 31 20:00:39.964146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.963541    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"]
Oct 31 20:00:39.964146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.963667    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:00:39.977369 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:39.977332    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbd28\" (UniqueName: \"kubernetes.io/projected/e7330358-d4c3-4f17-a61c-d794d4425a28-kube-api-access-rbd28\") pod \"payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj\" (UID: \"e7330358-d4c3-4f17-a61c-d794d4425a28\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:00:40.032495 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.032450    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:00:40.121674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.121606    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd\" (UID: \"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:00:40.121857 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.121789    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksdk\" (UniqueName: \"kubernetes.io/projected/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-kube-api-access-mksdk\") pod \"payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd\" (UID: \"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:00:40.222925 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.222829    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd\" (UID: \"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:00:40.223112 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.222959    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksdk\" (UniqueName: \"kubernetes.io/projected/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-kube-api-access-mksdk\") pod \"payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd\" (UID: \"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:00:40.223607 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.223580    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd\" (UID: \"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:00:40.282070 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.282033    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksdk\" (UniqueName: \"kubernetes.io/projected/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-kube-api-access-mksdk\") pod \"payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd\" (UID: \"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:00:40.430881 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.430850    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"]
Oct 31 20:00:40.497893 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.497858    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:40.581755 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.581713    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:00:40.628343 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.628308    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-xks-ca\") pod \"64204cf0-9632-49e2-8e2d-03612a4d6116\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") "
Oct 31 20:00:40.628554 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.628520    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64204cf0-9632-49e2-8e2d-03612a4d6116-kube-api-access-zx2g8\" (UniqueName: \"kubernetes.io/projected/64204cf0-9632-49e2-8e2d-03612a4d6116-kube-api-access-zx2g8\") pod \"64204cf0-9632-49e2-8e2d-03612a4d6116\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") "
Oct 31 20:00:40.628683 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.628572    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-openshift-ca\") pod \"64204cf0-9632-49e2-8e2d-03612a4d6116\" (UID: \"64204cf0-9632-49e2-8e2d-03612a4d6116\") "
Oct 31 20:00:40.629407 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.629379    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-openshift-ca" pod "64204cf0-9632-49e2-8e2d-03612a4d6116" (UID: "64204cf0-9632-49e2-8e2d-03612a4d6116"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:40.633424 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.631003    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64204cf0-9632-49e2-8e2d-03612a4d6116-kube-api-access-zx2g8" pod "64204cf0-9632-49e2-8e2d-03612a4d6116" (UID: "64204cf0-9632-49e2-8e2d-03612a4d6116"). InnerVolumeSpecName "kube-api-access-zx2g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:40.633839 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.633814    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-xks-ca" pod "64204cf0-9632-49e2-8e2d-03612a4d6116" (UID: "64204cf0-9632-49e2-8e2d-03612a4d6116"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:40.731083 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.731045    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx2g8\" (UniqueName: \"kubernetes.io/projected/64204cf0-9632-49e2-8e2d-03612a4d6116-kube-api-access-zx2g8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:40.731083 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.731084    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:40.731083 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.731099    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64204cf0-9632-49e2-8e2d-03612a4d6116-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:40.962438 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:00:40.962402    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd31ef1_4e8e_4ba4_aaa5_ec5e7697f1e6.slice/crio-837b14d21e53e6a0d121f93363d0c8de3a09aa1d2660cc4317b227cb81289af5 WatchSource:0}: Error finding container 837b14d21e53e6a0d121f93363d0c8de3a09aa1d2660cc4317b227cb81289af5: Status 404 returned error can't find the container with id 837b14d21e53e6a0d121f93363d0c8de3a09aa1d2660cc4317b227cb81289af5
Oct 31 20:00:40.965645 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:40.965615    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"]
Oct 31 20:00:41.349645 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.349602    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj" event={"ID":"e7330358-d4c3-4f17-a61c-d794d4425a28","Type":"ContainerStarted","Data":"6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee"}
Oct 31 20:00:41.349862 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.349749    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj" event={"ID":"e7330358-d4c3-4f17-a61c-d794d4425a28","Type":"ContainerStarted","Data":"5ebfcb943209444fd2bc8b4b4b9250f519f7eb6724a8ce7ace8e3ed658b42a78"}
Oct 31 20:00:41.349935 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.349905    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:00:41.351136 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.351116    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"
Oct 31 20:00:41.351270 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.351136    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr" event={"ID":"64204cf0-9632-49e2-8e2d-03612a4d6116","Type":"ContainerDied","Data":"502c044dd666cf88f10f34c912a3aaeedebc31fb6af1769364ef7c9b3f4c1bca"}
Oct 31 20:00:41.351270 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.351181    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="502c044dd666cf88f10f34c912a3aaeedebc31fb6af1769364ef7c9b3f4c1bca"
Oct 31 20:00:41.352805 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.352780    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd" event={"ID":"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6","Type":"ContainerStarted","Data":"236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e"}
Oct 31 20:00:41.352882 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.352820    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd" event={"ID":"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6","Type":"ContainerStarted","Data":"837b14d21e53e6a0d121f93363d0c8de3a09aa1d2660cc4317b227cb81289af5"}
Oct 31 20:00:41.352882 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.352840    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:00:41.464917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.464863    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj" podStartSLOduration=2.464850776 podStartE2EDuration="2.464850776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:41.464387742 +0000 UTC m=+2659.993921815" watchObservedRunningTime="2026-10-31 20:00:41.464850776 +0000 UTC m=+2659.994384829"
Oct 31 20:00:41.596608 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:41.592923    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd" podStartSLOduration=2.59290161 podStartE2EDuration="2.59290161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:41.581615718 +0000 UTC m=+2660.111149792" watchObservedRunningTime="2026-10-31 20:00:41.59290161 +0000 UTC m=+2660.122435663"
Oct 31 20:00:43.306976 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:43.306925    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:44.978054 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:44.978003    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:45.206930 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:45.206896    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:45.255024 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:45.254937    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"]
Oct 31 20:00:45.255223 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:45.255201    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy" containerID="cri-o://72d483d274e5f24c3ecdc7854a588eee66e68465599afa88c6a4beb69658699a" gracePeriod=30
Oct 31 20:00:45.536627 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:45.536537    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:46.039036 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.038994    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"]
Oct 31 20:00:46.039881 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.039857    5006 cpu_manager.go:414] "containerMap: removing container" podUID="64204cf0-9632-49e2-8e2d-03612a4d6116" containerName="revoke-keys"
Oct 31 20:00:46.039881 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.039880    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64204cf0-9632-49e2-8e2d-03612a4d6116" containerName="revoke-keys"
Oct 31 20:00:46.040062 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.040052    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="64204cf0-9632-49e2-8e2d-03612a4d6116" containerName="revoke-keys"
Oct 31 20:00:46.044902 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.044876    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.096873 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.096835    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"]
Oct 31 20:00:46.125427 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.125384    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.125622 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.125435    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.125622 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.125572    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm5qm\" (UniqueName: \"kubernetes.io/projected/c7474b55-95b2-425a-afae-92ffebc3d775-kube-api-access-nm5qm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.142719 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.142686    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:00:46.142719 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.142706    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:00:46.142940 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.142723    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:00:46.142940 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.142750    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:00:46.227109 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.227068    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm5qm\" (UniqueName: \"kubernetes.io/projected/c7474b55-95b2-425a-afae-92ffebc3d775-kube-api-access-nm5qm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.227297 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.227274    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.227297 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.227294    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.228025 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.228001    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.229725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.229688    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.284640 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.284605    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm5qm\" (UniqueName: \"kubernetes.io/projected/c7474b55-95b2-425a-afae-92ffebc3d775-kube-api-access-nm5qm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.360104 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.360072    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:46.737934 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:46.736593    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"]
Oct 31 20:00:47.111934 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:47.111906    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:00:47.438599 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:47.438559    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb" event={"ID":"c7474b55-95b2-425a-afae-92ffebc3d775","Type":"ContainerStarted","Data":"d10bbde5aa19d6bde0a332407c605a403a895f15a573a4ac7a0698eadedb3cec"}
Oct 31 20:00:47.438785 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:47.438618    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb" event={"ID":"c7474b55-95b2-425a-afae-92ffebc3d775","Type":"ContainerStarted","Data":"32bd7aad8d20fe303524488dd5a52f69cbf872a7310bf58a3140192f937cfdd2"}
Oct 31 20:00:47.556913 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:47.556848    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb" podStartSLOduration=2.556833634 podStartE2EDuration="2.556833634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:47.552825486 +0000 UTC m=+2666.082359578" watchObservedRunningTime="2026-10-31 20:00:47.556833634 +0000 UTC m=+2666.086367687"
Oct 31 20:00:48.446896 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:48.446855    5006 generic.go:356] "Generic (PLEG): container finished" podID="c7474b55-95b2-425a-afae-92ffebc3d775" containerID="d10bbde5aa19d6bde0a332407c605a403a895f15a573a4ac7a0698eadedb3cec" exitCode=0
Oct 31 20:00:48.446896 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:48.446891    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb" event={"ID":"c7474b55-95b2-425a-afae-92ffebc3d775","Type":"ContainerDied","Data":"d10bbde5aa19d6bde0a332407c605a403a895f15a573a4ac7a0698eadedb3cec"}
Oct 31 20:00:48.756713 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:48.756586    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:49.618489 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:49.618459    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:49.678042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:49.678006    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-openshift-ca\") pod \"c7474b55-95b2-425a-afae-92ffebc3d775\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") "
Oct 31 20:00:49.678223 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:49.678101    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7474b55-95b2-425a-afae-92ffebc3d775-kube-api-access-nm5qm\" (UniqueName: \"kubernetes.io/projected/c7474b55-95b2-425a-afae-92ffebc3d775-kube-api-access-nm5qm\") pod \"c7474b55-95b2-425a-afae-92ffebc3d775\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") "
Oct 31 20:00:49.678223 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:49.678185    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-xks-ca\") pod \"c7474b55-95b2-425a-afae-92ffebc3d775\" (UID: \"c7474b55-95b2-425a-afae-92ffebc3d775\") "
Oct 31 20:00:49.678368 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:49.678345    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-openshift-ca" pod "c7474b55-95b2-425a-afae-92ffebc3d775" (UID: "c7474b55-95b2-425a-afae-92ffebc3d775"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:49.678710 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:49.678685    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:49.680161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:49.680138    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7474b55-95b2-425a-afae-92ffebc3d775-kube-api-access-nm5qm" pod "c7474b55-95b2-425a-afae-92ffebc3d775" (UID: "c7474b55-95b2-425a-afae-92ffebc3d775"). InnerVolumeSpecName "kube-api-access-nm5qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:49.680241 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:49.680159    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-xks-ca" pod "c7474b55-95b2-425a-afae-92ffebc3d775" (UID: "c7474b55-95b2-425a-afae-92ffebc3d775"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:49.780557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:49.780450    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7474b55-95b2-425a-afae-92ffebc3d775-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:49.780557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:49.780493    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm5qm\" (UniqueName: \"kubernetes.io/projected/c7474b55-95b2-425a-afae-92ffebc3d775-kube-api-access-nm5qm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:50.464930 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.464878    5006 generic.go:356] "Generic (PLEG): container finished" podID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerID="72d483d274e5f24c3ecdc7854a588eee66e68465599afa88c6a4beb69658699a" exitCode=0
Oct 31 20:00:50.465126 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.465079    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" event={"ID":"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78","Type":"ContainerDied","Data":"72d483d274e5f24c3ecdc7854a588eee66e68465599afa88c6a4beb69658699a"}
Oct 31 20:00:50.465204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.465142    5006 scope.go:122] "RemoveContainer" containerID="69db270188f1bec2cd5b94539fa5301e1d35483d0ba12cb38451f232fe23eaae"
Oct 31 20:00:50.469224 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.469190    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb" event={"ID":"c7474b55-95b2-425a-afae-92ffebc3d775","Type":"ContainerDied","Data":"32bd7aad8d20fe303524488dd5a52f69cbf872a7310bf58a3140192f937cfdd2"}
Oct 31 20:00:50.469381 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.469254    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32bd7aad8d20fe303524488dd5a52f69cbf872a7310bf58a3140192f937cfdd2"
Oct 31 20:00:50.469381 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.469339    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"
Oct 31 20:00:50.535185 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.535153    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 20:00:50.592023 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.591991    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-socket\") pod \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") "
Oct 31 20:00:50.592023 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592041    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-podinfo\") pod \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") "
Oct 31 20:00:50.592287 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592063    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-credential-socket\") pod \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") "
Oct 31 20:00:50.592287 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592089    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-token\" (UniqueName: \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-token\") pod \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") "
Oct 31 20:00:50.592287 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592235    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-certs\") pod \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") "
Oct 31 20:00:50.592446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592290    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-kube-api-access-m8nqh\" (UniqueName: \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-kube-api-access-m8nqh\") pod \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") "
Oct 31 20:00:50.592446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592305    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-socket" pod "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" (UID: "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:50.592446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592365    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istiod-ca-cert\") pod \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") "
Oct 31 20:00:50.592446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592385    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-credential-socket" pod "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" (UID: "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:50.592446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592409    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-envoy\") pod \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") "
Oct 31 20:00:50.592446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592437    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-data\") pod \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\" (UID: \"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78\") "
Oct 31 20:00:50.592779 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592536    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-certs" pod "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" (UID: "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:50.592888 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592793    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istiod-ca-cert" pod "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" (UID: "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:50.592950 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.592852    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-data" pod "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" (UID: "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:50.593023 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.593006    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:50.593086 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.593031    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:50.593086 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.593047    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:50.593086 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.593059    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:50.593086 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.593071    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:50.594973 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.594946    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-kube-api-access-m8nqh" pod "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" (UID: "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78"). InnerVolumeSpecName "kube-api-access-m8nqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:50.594973 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.594962    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-podinfo" pod "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" (UID: "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:00:50.594973 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.594973    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-token" pod "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" (UID: "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:50.595172 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.594964    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-envoy" pod "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" (UID: "0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:50.694621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.694575    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:50.694621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.694617    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:50.695053 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.694633    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8nqh\" (UniqueName: \"kubernetes.io/projected/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-kube-api-access-m8nqh\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:50.695053 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:50.694666    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:51.477747 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:51.477702    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg" event={"ID":"0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78","Type":"ContainerDied","Data":"159dd88db34226588fc33c9dd850c04ccc891979f65985dd008717fb20656340"}
Oct 31 20:00:51.477747 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:51.477732    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"
Oct 31 20:00:51.478024 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:51.477778    5006 scope.go:122] "RemoveContainer" containerID="72d483d274e5f24c3ecdc7854a588eee66e68465599afa88c6a4beb69658699a"
Oct 31 20:00:51.646241 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:51.646204    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"]
Oct 31 20:00:51.696872 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:51.696838    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7c0d34-openshift-default-66cb596f7-9p9tg"]
Oct 31 20:00:52.368717 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:52.368645    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:00:52.376693 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:52.376620    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:00:53.600182 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:53.600142    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" path="/var/lib/kubelet/pods/0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78/volumes"
Oct 31 20:00:53.942609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:53.942512    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:58.003477 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:58.003437    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:58.211485 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:58.211449    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:59.228397 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:00:59.228351    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:00.352985 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:00.352943    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:00.578960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:00.578926    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:04.509674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:04.509483    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:05.092005 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.091964    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"]
Oct 31 20:01:05.092860 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.092839    5006 cpu_manager.go:414] "containerMap: removing container" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy"
Oct 31 20:01:05.092908 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.092863    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy"
Oct 31 20:01:05.092908 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.092903    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c7474b55-95b2-425a-afae-92ffebc3d775" containerName="revoke-keys"
Oct 31 20:01:05.092982 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.092909    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7474b55-95b2-425a-afae-92ffebc3d775" containerName="revoke-keys"
Oct 31 20:01:05.093013 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.093003    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy"
Oct 31 20:01:05.093049 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.093018    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7474b55-95b2-425a-afae-92ffebc3d775" containerName="revoke-keys"
Oct 31 20:01:05.098236 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.098215    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.150008 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.149969    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"]
Oct 31 20:01:05.195902 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.195824    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:01:05.195902 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.195842    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:01:05.195902 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.195874    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:01:05.218280 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.218233    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dfe658-2slfb\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.218477 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.218458    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2p4f\" (UniqueName: \"kubernetes.io/projected/c26abcc6-a5f1-48b8-b829-6a610e7307be-kube-api-access-v2p4f\") pod \"maas-api-revoke-keys-e2e-api-a-dfe658-2slfb\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.218540 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.218490    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dfe658-2slfb\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.228832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.228801    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:01:05.319391 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.319348    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2p4f\" (UniqueName: \"kubernetes.io/projected/c26abcc6-a5f1-48b8-b829-6a610e7307be-kube-api-access-v2p4f\") pod \"maas-api-revoke-keys-e2e-api-a-dfe658-2slfb\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.319591 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.319394    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dfe658-2slfb\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.319591 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.319522    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dfe658-2slfb\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.320210 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.320185    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dfe658-2slfb\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.322124 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.322094    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dfe658-2slfb\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.377245 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.377165    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2p4f\" (UniqueName: \"kubernetes.io/projected/c26abcc6-a5f1-48b8-b829-6a610e7307be-kube-api-access-v2p4f\") pod \"maas-api-revoke-keys-e2e-api-a-dfe658-2slfb\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.395806 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:01:05.395765    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:01:05.412878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.412844    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:05.787825 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:01:05.787782    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26abcc6_a5f1_48b8_b829_6a610e7307be.slice/crio-a9bff7e1b23d015ca6946df1ab409b06510cc5d017f93a0452866bdf2b018491 WatchSource:0}: Error finding container a9bff7e1b23d015ca6946df1ab409b06510cc5d017f93a0452866bdf2b018491: Status 404 returned error can't find the container with id a9bff7e1b23d015ca6946df1ab409b06510cc5d017f93a0452866bdf2b018491
Oct 31 20:01:05.790905 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:05.790879    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"]
Oct 31 20:01:06.584285 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:06.584194    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb" event={"ID":"c26abcc6-a5f1-48b8-b829-6a610e7307be","Type":"ContainerStarted","Data":"17af874350d04a2f4676b04ba8d83d909a81f744ebaab7bc52dcedc0cd2482cd"}
Oct 31 20:01:06.584540 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:06.584290    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb" event={"ID":"c26abcc6-a5f1-48b8-b829-6a610e7307be","Type":"ContainerStarted","Data":"a9bff7e1b23d015ca6946df1ab409b06510cc5d017f93a0452866bdf2b018491"}
Oct 31 20:01:06.697912 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:06.697860    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb" podStartSLOduration=1.697846729 podStartE2EDuration="1.697846729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:06.696952133 +0000 UTC m=+2685.226486198" watchObservedRunningTime="2026-10-31 20:01:06.697846729 +0000 UTC m=+2685.227380782"
Oct 31 20:01:07.593088 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:07.593054    5006 generic.go:356] "Generic (PLEG): container finished" podID="c26abcc6-a5f1-48b8-b829-6a610e7307be" containerID="17af874350d04a2f4676b04ba8d83d909a81f744ebaab7bc52dcedc0cd2482cd" exitCode=0
Oct 31 20:01:07.599546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:07.599507    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb" event={"ID":"c26abcc6-a5f1-48b8-b829-6a610e7307be","Type":"ContainerDied","Data":"17af874350d04a2f4676b04ba8d83d909a81f744ebaab7bc52dcedc0cd2482cd"}
Oct 31 20:01:08.767383 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:08.767355    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:08.865931 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:08.865834    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-xks-ca\") pod \"c26abcc6-a5f1-48b8-b829-6a610e7307be\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") "
Oct 31 20:01:08.865931 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:08.865878    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26abcc6-a5f1-48b8-b829-6a610e7307be-kube-api-access-v2p4f\" (UniqueName: \"kubernetes.io/projected/c26abcc6-a5f1-48b8-b829-6a610e7307be-kube-api-access-v2p4f\") pod \"c26abcc6-a5f1-48b8-b829-6a610e7307be\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") "
Oct 31 20:01:08.866142 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:08.865971    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-openshift-ca\") pod \"c26abcc6-a5f1-48b8-b829-6a610e7307be\" (UID: \"c26abcc6-a5f1-48b8-b829-6a610e7307be\") "
Oct 31 20:01:08.866338 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:08.866314    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-openshift-ca" pod "c26abcc6-a5f1-48b8-b829-6a610e7307be" (UID: "c26abcc6-a5f1-48b8-b829-6a610e7307be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:08.866618 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:08.866600    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:08.868116 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:08.868092    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26abcc6-a5f1-48b8-b829-6a610e7307be-kube-api-access-v2p4f" pod "c26abcc6-a5f1-48b8-b829-6a610e7307be" (UID: "c26abcc6-a5f1-48b8-b829-6a610e7307be"). InnerVolumeSpecName "kube-api-access-v2p4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:08.868240 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:08.868120    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-xks-ca" pod "c26abcc6-a5f1-48b8-b829-6a610e7307be" (UID: "c26abcc6-a5f1-48b8-b829-6a610e7307be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:08.968450 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:08.968412    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26abcc6-a5f1-48b8-b829-6a610e7307be-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:08.968450 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:08.968447    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2p4f\" (UniqueName: \"kubernetes.io/projected/c26abcc6-a5f1-48b8-b829-6a610e7307be-kube-api-access-v2p4f\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:09.609216 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:09.609194    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"
Oct 31 20:01:09.609331 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:09.609301    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb" event={"ID":"c26abcc6-a5f1-48b8-b829-6a610e7307be","Type":"ContainerDied","Data":"a9bff7e1b23d015ca6946df1ab409b06510cc5d017f93a0452866bdf2b018491"}
Oct 31 20:01:09.609416 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:09.609360    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9bff7e1b23d015ca6946df1ab409b06510cc5d017f93a0452866bdf2b018491"
Oct 31 20:01:09.783870 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:09.783831    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:15.013960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:15.013864    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:17.756179 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:17.756135    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:17.972121 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:17.972085    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:19.512783 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:19.512743    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:20.726800 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:20.726763    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:22.141401 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:22.141373    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:01:22.228789 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:22.228761    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:01:22.520692 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:22.520596    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:22.778753 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:22.778616    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:23.000489 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:23.000454    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:23.017498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:23.017461    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"]
Oct 31 20:01:23.067424 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:23.067338    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a5ba62-bqqgn"]
Oct 31 20:01:23.443531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:23.443498    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:23.599041 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:23.599006    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf" path="/var/lib/kubelet/pods/dc1d7bf0-ea3e-4e6c-a0d3-f21bcb6d19bf/volumes"
Oct 31 20:01:24.853378 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.853341    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"]
Oct 31 20:01:24.853966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.853952    5006 cpu_manager.go:414] "containerMap: removing container" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy"
Oct 31 20:01:24.854019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.853968    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy"
Oct 31 20:01:24.854019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.854004    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c26abcc6-a5f1-48b8-b829-6a610e7307be" containerName="revoke-keys"
Oct 31 20:01:24.854019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.854009    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26abcc6-a5f1-48b8-b829-6a610e7307be" containerName="revoke-keys"
Oct 31 20:01:24.854130 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.854114    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8c5d8a-bd52-4cfa-ac40-b32b8fbb3d78" containerName="istio-proxy"
Oct 31 20:01:24.854167 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.854136    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26abcc6-a5f1-48b8-b829-6a610e7307be" containerName="revoke-keys"
Oct 31 20:01:24.859404 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.859381    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:24.913170 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.913131    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"]
Oct 31 20:01:24.947216 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.947171    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:24.947436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.947412    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:24.947514 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.947494    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpm2b\" (UniqueName: \"kubernetes.io/projected/79e6c134-a769-49d0-a220-40c8a0d27a94-kube-api-access-mpm2b\") pod \"maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:24.959198 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.959167    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:01:24.963208 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.963177    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:01:24.963369 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.963206    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:01:24.963369 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:24.963224    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:01:25.048400 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:25.048360    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:25.048599 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:25.048426    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpm2b\" (UniqueName: \"kubernetes.io/projected/79e6c134-a769-49d0-a220-40c8a0d27a94-kube-api-access-mpm2b\") pod \"maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:25.048599 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:25.048530    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:25.049284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:25.049261    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:25.051376 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:25.051357    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:25.105917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:25.105835    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpm2b\" (UniqueName: \"kubernetes.io/projected/79e6c134-a769-49d0-a220-40c8a0d27a94-kube-api-access-mpm2b\") pod \"maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:25.174893 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:25.174849    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:25.548323 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:01:25.548284    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e6c134_a769_49d0_a220_40c8a0d27a94.slice/crio-baa5bfa259117ce187611a71a3705df43b1734f9a32e2880b71c1b3a7a1e2ad8 WatchSource:0}: Error finding container baa5bfa259117ce187611a71a3705df43b1734f9a32e2880b71c1b3a7a1e2ad8: Status 404 returned error can't find the container with id baa5bfa259117ce187611a71a3705df43b1734f9a32e2880b71c1b3a7a1e2ad8
Oct 31 20:01:25.551147 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:25.551123    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"]
Oct 31 20:01:25.809822 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:25.809784    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7" event={"ID":"79e6c134-a769-49d0-a220-40c8a0d27a94","Type":"ContainerStarted","Data":"baa5bfa259117ce187611a71a3705df43b1734f9a32e2880b71c1b3a7a1e2ad8"}
Oct 31 20:01:26.818293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:26.818077    5006 generic.go:356] "Generic (PLEG): container finished" podID="79e6c134-a769-49d0-a220-40c8a0d27a94" containerID="2e16a94b06aec8de7982048691c1292cad9d14b40bcb34ccea36e3de02eb3ea1" exitCode=0
Oct 31 20:01:26.818293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:26.818168    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7" event={"ID":"79e6c134-a769-49d0-a220-40c8a0d27a94","Type":"ContainerDied","Data":"2e16a94b06aec8de7982048691c1292cad9d14b40bcb34ccea36e3de02eb3ea1"}
Oct 31 20:01:26.885988 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:26.885952    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:27.915309 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:27.915271    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:28.020918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.020877    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:28.096275 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.096243    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e6c134-a769-49d0-a220-40c8a0d27a94-kube-api-access-mpm2b\" (UniqueName: \"kubernetes.io/projected/79e6c134-a769-49d0-a220-40c8a0d27a94-kube-api-access-mpm2b\") pod \"79e6c134-a769-49d0-a220-40c8a0d27a94\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") "
Oct 31 20:01:28.096442 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.096358    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-openshift-ca\") pod \"79e6c134-a769-49d0-a220-40c8a0d27a94\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") "
Oct 31 20:01:28.096442 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.096410    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-xks-ca\") pod \"79e6c134-a769-49d0-a220-40c8a0d27a94\" (UID: \"79e6c134-a769-49d0-a220-40c8a0d27a94\") "
Oct 31 20:01:28.096786 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.096757    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-openshift-ca" pod "79e6c134-a769-49d0-a220-40c8a0d27a94" (UID: "79e6c134-a769-49d0-a220-40c8a0d27a94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:28.097021 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.096998    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:28.098632 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.098607    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-xks-ca" pod "79e6c134-a769-49d0-a220-40c8a0d27a94" (UID: "79e6c134-a769-49d0-a220-40c8a0d27a94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:28.098754 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.098641    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e6c134-a769-49d0-a220-40c8a0d27a94-kube-api-access-mpm2b" pod "79e6c134-a769-49d0-a220-40c8a0d27a94" (UID: "79e6c134-a769-49d0-a220-40c8a0d27a94"). InnerVolumeSpecName "kube-api-access-mpm2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:28.198377 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.198344    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e6c134-a769-49d0-a220-40c8a0d27a94-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:28.198377 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.198375    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpm2b\" (UniqueName: \"kubernetes.io/projected/79e6c134-a769-49d0-a220-40c8a0d27a94-kube-api-access-mpm2b\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:28.833986 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.833954    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"
Oct 31 20:01:28.834175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.833979    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7" event={"ID":"79e6c134-a769-49d0-a220-40c8a0d27a94","Type":"ContainerDied","Data":"baa5bfa259117ce187611a71a3705df43b1734f9a32e2880b71c1b3a7a1e2ad8"}
Oct 31 20:01:28.834175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:28.834029    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baa5bfa259117ce187611a71a3705df43b1734f9a32e2880b71c1b3a7a1e2ad8"
Oct 31 20:01:29.396668 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:29.396628    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:29.625293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:29.625259    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:29.853752 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:29.852912    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:30.475399 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.475356    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"]
Oct 31 20:01:30.476025 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.476009    5006 cpu_manager.go:414] "containerMap: removing container" podUID="79e6c134-a769-49d0-a220-40c8a0d27a94" containerName="revoke-keys"
Oct 31 20:01:30.476071 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.476026    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e6c134-a769-49d0-a220-40c8a0d27a94" containerName="revoke-keys"
Oct 31 20:01:30.476177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.476163    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e6c134-a769-49d0-a220-40c8a0d27a94" containerName="revoke-keys"
Oct 31 20:01:30.480793 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.480765    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:30.535096 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.535055    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"]
Oct 31 20:01:30.538404 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.538371    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:30.538596 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.538428    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd67z\" (UniqueName: \"kubernetes.io/projected/8c4add7c-97cb-4e50-8619-7809d82ff0f7-kube-api-access-nd67z\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:30.538596 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.538537    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:30.579527 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.579492    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:01:30.579799 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.579565    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:01:30.583625 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.583598    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:01:30.583625 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.583616    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:01:30.640010 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.639965    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:30.640210 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.640030    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd67z\" (UniqueName: \"kubernetes.io/projected/8c4add7c-97cb-4e50-8619-7809d82ff0f7-kube-api-access-nd67z\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:30.640210 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.640077    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:30.640966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.640940    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:30.642715 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.642693    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:30.697914 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.697870    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd67z\" (UniqueName: \"kubernetes.io/projected/8c4add7c-97cb-4e50-8619-7809d82ff0f7-kube-api-access-nd67z\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:30.798838 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:30.798752    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:31.209372 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:31.209121    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"]
Oct 31 20:01:31.874582 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:31.874525    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7" event={"ID":"8c4add7c-97cb-4e50-8619-7809d82ff0f7","Type":"ContainerStarted","Data":"9bce21e1b04b0400afc0991c7a2d45c60a5ec5ca9172b0b28b8a9f2a9e91a16d"}
Oct 31 20:01:31.875145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:31.874598    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7" event={"ID":"8c4add7c-97cb-4e50-8619-7809d82ff0f7","Type":"ContainerStarted","Data":"f4485b67c562e99345184d937af059c3fb2c16d43dd43bb9bfc6ac48765d0752"}
Oct 31 20:01:32.006181 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:32.006107    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7" podStartSLOduration=2.006083543 podStartE2EDuration="2.006083543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:31.99910132 +0000 UTC m=+2710.528635376" watchObservedRunningTime="2026-10-31 20:01:32.006083543 +0000 UTC m=+2710.535617599"
Oct 31 20:01:32.884603 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:32.884560    5006 generic.go:356] "Generic (PLEG): container finished" podID="8c4add7c-97cb-4e50-8619-7809d82ff0f7" containerID="9bce21e1b04b0400afc0991c7a2d45c60a5ec5ca9172b0b28b8a9f2a9e91a16d" exitCode=0
Oct 31 20:01:32.885103 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:32.884705    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7" event={"ID":"8c4add7c-97cb-4e50-8619-7809d82ff0f7","Type":"ContainerDied","Data":"9bce21e1b04b0400afc0991c7a2d45c60a5ec5ca9172b0b28b8a9f2a9e91a16d"}
Oct 31 20:01:33.323161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:33.323068    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:34.064427 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.064397    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:34.198960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.198845    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-openshift-ca\") pod \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") "
Oct 31 20:01:34.198960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.198945    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4add7c-97cb-4e50-8619-7809d82ff0f7-kube-api-access-nd67z\" (UniqueName: \"kubernetes.io/projected/8c4add7c-97cb-4e50-8619-7809d82ff0f7-kube-api-access-nd67z\") pod \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") "
Oct 31 20:01:34.199216 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.199082    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-xks-ca\") pod \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\" (UID: \"8c4add7c-97cb-4e50-8619-7809d82ff0f7\") "
Oct 31 20:01:34.200145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.200088    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-openshift-ca" pod "8c4add7c-97cb-4e50-8619-7809d82ff0f7" (UID: "8c4add7c-97cb-4e50-8619-7809d82ff0f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:34.202594 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.202563    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-xks-ca" pod "8c4add7c-97cb-4e50-8619-7809d82ff0f7" (UID: "8c4add7c-97cb-4e50-8619-7809d82ff0f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:34.203052 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.203034    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4add7c-97cb-4e50-8619-7809d82ff0f7-kube-api-access-nd67z" pod "8c4add7c-97cb-4e50-8619-7809d82ff0f7" (UID: "8c4add7c-97cb-4e50-8619-7809d82ff0f7"). InnerVolumeSpecName "kube-api-access-nd67z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:34.300916 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.300880    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:34.300916 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.300913    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4add7c-97cb-4e50-8619-7809d82ff0f7-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:34.301148 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.300928    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd67z\" (UniqueName: \"kubernetes.io/projected/8c4add7c-97cb-4e50-8619-7809d82ff0f7-kube-api-access-nd67z\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:34.901105 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.901074    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"
Oct 31 20:01:34.901285 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.901107    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7" event={"ID":"8c4add7c-97cb-4e50-8619-7809d82ff0f7","Type":"ContainerDied","Data":"f4485b67c562e99345184d937af059c3fb2c16d43dd43bb9bfc6ac48765d0752"}
Oct 31 20:01:34.901285 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:34.901153    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4485b67c562e99345184d937af059c3fb2c16d43dd43bb9bfc6ac48765d0752"
Oct 31 20:01:35.394511 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:01:35.394470    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:01:36.817363 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:36.817282    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:37.039482 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:37.039451    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:38.578470 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:38.578432    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:43.054162 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:43.054126    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:43.269563 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:43.269501    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:45.068985 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:45.068953    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:45.355802 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:45.355761    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:45.538687 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:45.538633    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"]
Oct 31 20:01:45.566059 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:45.566022    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:45.589035 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:45.588998    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-1de410-8cvm2"]
Oct 31 20:01:45.599468 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:45.599419    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35970881-0106-4496-ad86-d908337d2c7c" path="/var/lib/kubelet/pods/35970881-0106-4496-ad86-d908337d2c7c/volumes"
Oct 31 20:01:45.997098 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:45.997059    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:50.767743 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:50.767699    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:52.052213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:52.052176    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:52.287899 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:52.287859    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:52.525521 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:52.525480    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:53.125995 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.125955    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"]
Oct 31 20:01:53.126619 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.126598    5006 cpu_manager.go:414] "containerMap: removing container" podUID="8c4add7c-97cb-4e50-8619-7809d82ff0f7" containerName="revoke-keys"
Oct 31 20:01:53.126686 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.126620    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4add7c-97cb-4e50-8619-7809d82ff0f7" containerName="revoke-keys"
Oct 31 20:01:53.126779 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.126769    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4add7c-97cb-4e50-8619-7809d82ff0f7" containerName="revoke-keys"
Oct 31 20:01:53.130657 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.130626    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.186104 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.186066    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"]
Oct 31 20:01:53.232187 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.232148    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:01:53.232381 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.232202    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:01:53.232381 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.232218    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:01:53.233974 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.233959    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:01:53.243410 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.243363    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6vm\" (UniqueName: \"kubernetes.io/projected/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-kube-api-access-dk6vm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.243585 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.243495    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.243585 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.243527    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.345474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.345424    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6vm\" (UniqueName: \"kubernetes.io/projected/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-kube-api-access-dk6vm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.345729 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.345495    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.345729 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.345516    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.346236 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.346213    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.348057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.348023    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.404145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.404063    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6vm\" (UniqueName: \"kubernetes.io/projected/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-kube-api-access-dk6vm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.445969 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.445931    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:53.818620 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:01:53.818571    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e76037_9673_4b7b_bd7d_1b12bc1d6abd.slice/crio-74a0368021e3a7fa580fff2052a1b316904770158f73154de54788ec43d6041e WatchSource:0}: Error finding container 74a0368021e3a7fa580fff2052a1b316904770158f73154de54788ec43d6041e: Status 404 returned error can't find the container with id 74a0368021e3a7fa580fff2052a1b316904770158f73154de54788ec43d6041e
Oct 31 20:01:53.822124 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:53.822100    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"]
Oct 31 20:01:54.077366 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:54.077266    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5" event={"ID":"27e76037-9673-4b7b-bd7d-1b12bc1d6abd","Type":"ContainerStarted","Data":"74a0368021e3a7fa580fff2052a1b316904770158f73154de54788ec43d6041e"}
Oct 31 20:01:55.088126 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:55.088091    5006 generic.go:356] "Generic (PLEG): container finished" podID="27e76037-9673-4b7b-bd7d-1b12bc1d6abd" containerID="8c8b6eaae6a925460f6ed5e8c02f72fb9e8e7fd25c2b19f14a209cedc299d201" exitCode=0
Oct 31 20:01:55.088561 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:55.088218    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5" event={"ID":"27e76037-9673-4b7b-bd7d-1b12bc1d6abd","Type":"ContainerDied","Data":"8c8b6eaae6a925460f6ed5e8c02f72fb9e8e7fd25c2b19f14a209cedc299d201"}
Oct 31 20:01:55.386704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:55.386604    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:01:55.909853 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:55.909801    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:56.273221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:56.273189    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:56.438702 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:56.438130    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-kube-api-access-dk6vm\" (UniqueName: \"kubernetes.io/projected/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-kube-api-access-dk6vm\") pod \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") "
Oct 31 20:01:56.438702 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:56.438244    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-xks-ca\") pod \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") "
Oct 31 20:01:56.441939 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:56.441900    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-openshift-ca\") pod \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\" (UID: \"27e76037-9673-4b7b-bd7d-1b12bc1d6abd\") "
Oct 31 20:01:56.472313 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:56.471946    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-openshift-ca" pod "27e76037-9673-4b7b-bd7d-1b12bc1d6abd" (UID: "27e76037-9673-4b7b-bd7d-1b12bc1d6abd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:56.510133 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:56.510015    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-kube-api-access-dk6vm" pod "27e76037-9673-4b7b-bd7d-1b12bc1d6abd" (UID: "27e76037-9673-4b7b-bd7d-1b12bc1d6abd"). InnerVolumeSpecName "kube-api-access-dk6vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:56.517688 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:56.514362    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-xks-ca" pod "27e76037-9673-4b7b-bd7d-1b12bc1d6abd" (UID: "27e76037-9673-4b7b-bd7d-1b12bc1d6abd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:56.543900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:56.543861    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk6vm\" (UniqueName: \"kubernetes.io/projected/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-kube-api-access-dk6vm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:56.543900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:56.543895    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:56.543900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:56.543909    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e76037-9673-4b7b-bd7d-1b12bc1d6abd-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:57.103938 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:57.103903    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"
Oct 31 20:01:57.103938 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:57.103933    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5" event={"ID":"27e76037-9673-4b7b-bd7d-1b12bc1d6abd","Type":"ContainerDied","Data":"74a0368021e3a7fa580fff2052a1b316904770158f73154de54788ec43d6041e"}
Oct 31 20:01:57.104170 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:01:57.103985    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74a0368021e3a7fa580fff2052a1b316904770158f73154de54788ec43d6041e"
Oct 31 20:02:01.016770 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:01.016633    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:01.228259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:01.228226    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:05.075010 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:05.074974    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:05.300368 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:05.300333    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:05.398005 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:02:05.397974    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:02:05.899973 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:05.899938    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:07.279456 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:07.279422    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:07.704987 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:07.704945    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:08.000623 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:08.000537    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:08.150224 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:08.150190    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"]
Oct 31 20:02:08.199432 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:08.199401    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h4jkq"]
Oct 31 20:02:09.013050 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:09.013014    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:09.282001 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:09.281886    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:09.430419 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:09.430385    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"]
Oct 31 20:02:09.481101 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:09.481060    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-9a8346-k6fgb"]
Oct 31 20:02:09.503444 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:09.503411    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:09.600018 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:09.599978    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fcb3780-5dd3-46cc-83ad-2107c71256a9" path="/var/lib/kubelet/pods/2fcb3780-5dd3-46cc-83ad-2107c71256a9/volumes"
Oct 31 20:02:09.600489 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:09.600473    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7474b55-95b2-425a-afae-92ffebc3d775" path="/var/lib/kubelet/pods/c7474b55-95b2-425a-afae-92ffebc3d775/volumes"
Oct 31 20:02:09.923637 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:09.923556    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:11.649239 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:11.649191    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:11.864498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:11.864462    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:14.633832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:14.633797    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:15.605070 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:15.605035    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:16.128540 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:16.128500    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:16.364221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:16.364187    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:16.584156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:16.584071    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:17.054073 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.054036    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"]
Oct 31 20:02:17.054329 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.054288    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr" podUID="476b810f-d23d-475f-be1c-2d08b360a899" containerName="discovery" containerID="cri-o://1d60b1cc0d098a3a8b0387fb10367f9f3f3d50a7ae7705ab64c052fdcd5f62e5" gracePeriod=30
Oct 31 20:02:17.273126 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.273082    5006 generic.go:356] "Generic (PLEG): container finished" podID="476b810f-d23d-475f-be1c-2d08b360a899" containerID="1d60b1cc0d098a3a8b0387fb10367f9f3f3d50a7ae7705ab64c052fdcd5f62e5" exitCode=0
Oct 31 20:02:17.273739 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.273161    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr" event={"ID":"476b810f-d23d-475f-be1c-2d08b360a899","Type":"ContainerDied","Data":"1d60b1cc0d098a3a8b0387fb10367f9f3f3d50a7ae7705ab64c052fdcd5f62e5"}
Oct 31 20:02:17.372229 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.372196    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:17.375328 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.375306    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 20:02:17.486896 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.486848    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-ca-configmap\") pod \"476b810f-d23d-475f-be1c-2d08b360a899\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") "
Oct 31 20:02:17.487100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.487011    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476b810f-d23d-475f-be1c-2d08b360a899-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/476b810f-d23d-475f-be1c-2d08b360a899-local-certs\") pod \"476b810f-d23d-475f-be1c-2d08b360a899\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") "
Oct 31 20:02:17.487100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.487092    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-cacerts\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-cacerts\") pod \"476b810f-d23d-475f-be1c-2d08b360a899\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") "
Oct 31 20:02:17.487199 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.487128    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-kube-api-access-j7jbk\" (UniqueName: \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-kube-api-access-j7jbk\") pod \"476b810f-d23d-475f-be1c-2d08b360a899\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") "
Oct 31 20:02:17.487199 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.487176    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-kubeconfig\") pod \"476b810f-d23d-475f-be1c-2d08b360a899\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") "
Oct 31 20:02:17.487306 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.487257    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-ca-configmap" pod "476b810f-d23d-475f-be1c-2d08b360a899" (UID: "476b810f-d23d-475f-be1c-2d08b360a899"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:17.487306 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.487277    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-istio-token\" (UniqueName: \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-istio-token\") pod \"476b810f-d23d-475f-be1c-2d08b360a899\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") "
Oct 31 20:02:17.487425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.487329    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-dns-cert\") pod \"476b810f-d23d-475f-be1c-2d08b360a899\" (UID: \"476b810f-d23d-475f-be1c-2d08b360a899\") "
Oct 31 20:02:17.487996 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.487966    5006 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-ca-configmap\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:17.489420 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.489391    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476b810f-d23d-475f-be1c-2d08b360a899-local-certs" pod "476b810f-d23d-475f-be1c-2d08b360a899" (UID: "476b810f-d23d-475f-be1c-2d08b360a899"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:17.489552 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.489467    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-cacerts" pod "476b810f-d23d-475f-be1c-2d08b360a899" (UID: "476b810f-d23d-475f-be1c-2d08b360a899"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:17.489755 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.489732    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-kubeconfig" pod "476b810f-d23d-475f-be1c-2d08b360a899" (UID: "476b810f-d23d-475f-be1c-2d08b360a899"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:17.489905 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.489886    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-istio-token" pod "476b810f-d23d-475f-be1c-2d08b360a899" (UID: "476b810f-d23d-475f-be1c-2d08b360a899"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:17.489965 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.489915    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-kube-api-access-j7jbk" pod "476b810f-d23d-475f-be1c-2d08b360a899" (UID: "476b810f-d23d-475f-be1c-2d08b360a899"). InnerVolumeSpecName "kube-api-access-j7jbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:17.489965 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.489946    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-dns-cert" pod "476b810f-d23d-475f-be1c-2d08b360a899" (UID: "476b810f-d23d-475f-be1c-2d08b360a899"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:17.588923 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.588816    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7jbk\" (UniqueName: \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-kube-api-access-j7jbk\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:17.588923 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.588856    5006 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-kubeconfig\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:17.588923 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.588867    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476b810f-d23d-475f-be1c-2d08b360a899-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:17.588923 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.588875    5006 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-istio-csr-dns-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:17.588923 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.588884    5006 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/476b810f-d23d-475f-be1c-2d08b360a899-local-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:17.588923 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.588892    5006 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/476b810f-d23d-475f-be1c-2d08b360a899-cacerts\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:17.835843 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.835808    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"]
Oct 31 20:02:17.836417 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.836403    5006 cpu_manager.go:414] "containerMap: removing container" podUID="476b810f-d23d-475f-be1c-2d08b360a899" containerName="discovery"
Oct 31 20:02:17.836467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.836419    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476b810f-d23d-475f-be1c-2d08b360a899" containerName="discovery"
Oct 31 20:02:17.836467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.836436    5006 cpu_manager.go:414] "containerMap: removing container" podUID="27e76037-9673-4b7b-bd7d-1b12bc1d6abd" containerName="revoke-keys"
Oct 31 20:02:17.836467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.836441    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e76037-9673-4b7b-bd7d-1b12bc1d6abd" containerName="revoke-keys"
Oct 31 20:02:17.836562 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.836543    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="476b810f-d23d-475f-be1c-2d08b360a899" containerName="discovery"
Oct 31 20:02:17.836562 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.836551    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e76037-9673-4b7b-bd7d-1b12bc1d6abd" containerName="revoke-keys"
Oct 31 20:02:17.840445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.840376    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:17.893324 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.893290    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:02:17.893481 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.893370    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:02:17.893481 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.893301    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:02:17.893595 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.893515    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:02:17.894608 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.894588    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"]
Oct 31 20:02:17.995616 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.995580    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:17.995616 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.995625    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2dkv\" (UniqueName: \"kubernetes.io/projected/78899e07-f80a-4af7-90bf-2be049d010ee-kube-api-access-k2dkv\") pod \"maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:17.995900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:17.995875    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:18.097731 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.097590    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:18.097731 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.097639    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2dkv\" (UniqueName: \"kubernetes.io/projected/78899e07-f80a-4af7-90bf-2be049d010ee-kube-api-access-k2dkv\") pod \"maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:18.097991 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.097798    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:18.098485 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.098461    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:18.100224 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.100198    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:18.155214 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.155178    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2dkv\" (UniqueName: \"kubernetes.io/projected/78899e07-f80a-4af7-90bf-2be049d010ee-kube-api-access-k2dkv\") pod \"maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:18.282546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.282516    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"
Oct 31 20:02:18.283007 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.282548    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr" event={"ID":"476b810f-d23d-475f-be1c-2d08b360a899","Type":"ContainerDied","Data":"a061628bf44717715123fd5d254fe03c8a294bb44f0845581d56c6b0037a15f4"}
Oct 31 20:02:18.283007 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.282617    5006 scope.go:122] "RemoveContainer" containerID="1d60b1cc0d098a3a8b0387fb10367f9f3f3d50a7ae7705ab64c052fdcd5f62e5"
Oct 31 20:02:18.450191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.450087    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"]
Oct 31 20:02:18.454203 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.454176    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:18.456188 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.456161    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xg4nr"]
Oct 31 20:02:18.834299 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:02:18.834261    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78899e07_f80a_4af7_90bf_2be049d010ee.slice/crio-8f84a58de7ce41cb0ad878cb08e16083c857936557c3b40fe496f94699ecbd50 WatchSource:0}: Error finding container 8f84a58de7ce41cb0ad878cb08e16083c857936557c3b40fe496f94699ecbd50: Status 404 returned error can't find the container with id 8f84a58de7ce41cb0ad878cb08e16083c857936557c3b40fe496f94699ecbd50
Oct 31 20:02:18.837399 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:18.837369    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"]
Oct 31 20:02:19.292966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:19.292920    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v" event={"ID":"78899e07-f80a-4af7-90bf-2be049d010ee","Type":"ContainerStarted","Data":"129d0b684529927e1af6df32a55eb979dc0044bb9728c1b13ebf6dba291e9086"}
Oct 31 20:02:19.293408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:19.292988    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v" event={"ID":"78899e07-f80a-4af7-90bf-2be049d010ee","Type":"ContainerStarted","Data":"8f84a58de7ce41cb0ad878cb08e16083c857936557c3b40fe496f94699ecbd50"}
Oct 31 20:02:19.408347 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:19.408213    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v" podStartSLOduration=2.408193126 podStartE2EDuration="2.408193126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:19.40691023 +0000 UTC m=+2757.936444284" watchObservedRunningTime="2026-10-31 20:02:19.408193126 +0000 UTC m=+2757.937727179"
Oct 31 20:02:19.521977 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:19.521945    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:19.601017 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:19.600979    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476b810f-d23d-475f-be1c-2d08b360a899" path="/var/lib/kubelet/pods/476b810f-d23d-475f-be1c-2d08b360a899/volumes"
Oct 31 20:02:19.733055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:19.732965    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:20.018151 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:20.018068    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:20.302446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:20.302340    5006 generic.go:356] "Generic (PLEG): container finished" podID="78899e07-f80a-4af7-90bf-2be049d010ee" containerID="129d0b684529927e1af6df32a55eb979dc0044bb9728c1b13ebf6dba291e9086" exitCode=0
Oct 31 20:02:20.302446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:20.302425    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v" event={"ID":"78899e07-f80a-4af7-90bf-2be049d010ee","Type":"ContainerDied","Data":"129d0b684529927e1af6df32a55eb979dc0044bb9728c1b13ebf6dba291e9086"}
Oct 31 20:02:20.673993 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:20.673945    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:20.900552 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:20.900511    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:21.118958 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.118924    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:21.441113 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.441076    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:21.482576 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.482545    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:21.541253 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.541215    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-openshift-ca\") pod \"78899e07-f80a-4af7-90bf-2be049d010ee\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") "
Oct 31 20:02:21.541446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.541276    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-xks-ca\") pod \"78899e07-f80a-4af7-90bf-2be049d010ee\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") "
Oct 31 20:02:21.541492 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.541451    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78899e07-f80a-4af7-90bf-2be049d010ee-kube-api-access-k2dkv\" (UniqueName: \"kubernetes.io/projected/78899e07-f80a-4af7-90bf-2be049d010ee-kube-api-access-k2dkv\") pod \"78899e07-f80a-4af7-90bf-2be049d010ee\" (UID: \"78899e07-f80a-4af7-90bf-2be049d010ee\") "
Oct 31 20:02:21.541624 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.541601    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-openshift-ca" pod "78899e07-f80a-4af7-90bf-2be049d010ee" (UID: "78899e07-f80a-4af7-90bf-2be049d010ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:21.541981 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.541959    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:21.543584 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.543553    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-xks-ca" pod "78899e07-f80a-4af7-90bf-2be049d010ee" (UID: "78899e07-f80a-4af7-90bf-2be049d010ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:21.543722 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.543615    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78899e07-f80a-4af7-90bf-2be049d010ee-kube-api-access-k2dkv" pod "78899e07-f80a-4af7-90bf-2be049d010ee" (UID: "78899e07-f80a-4af7-90bf-2be049d010ee"). InnerVolumeSpecName "kube-api-access-k2dkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:21.643127 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.643017    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2dkv\" (UniqueName: \"kubernetes.io/projected/78899e07-f80a-4af7-90bf-2be049d010ee-kube-api-access-k2dkv\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:21.643127 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.643056    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78899e07-f80a-4af7-90bf-2be049d010ee-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:21.857311 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:21.857263    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:22.074003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.073729    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:22.258880 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.258844    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"]
Oct 31 20:02:22.269149 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.269113    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"]
Oct 31 20:02:22.318416 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.318381    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v" event={"ID":"78899e07-f80a-4af7-90bf-2be049d010ee","Type":"ContainerDied","Data":"8f84a58de7ce41cb0ad878cb08e16083c857936557c3b40fe496f94699ecbd50"}
Oct 31 20:02:22.318622 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.318460    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f84a58de7ce41cb0ad878cb08e16083c857936557c3b40fe496f94699ecbd50"
Oct 31 20:02:22.318622 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.318395    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"
Oct 31 20:02:22.335169 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.335087    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"]
Oct 31 20:02:22.335704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.335689    5006 cpu_manager.go:414] "containerMap: removing container" podUID="78899e07-f80a-4af7-90bf-2be049d010ee" containerName="revoke-keys"
Oct 31 20:02:22.335769 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.335705    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78899e07-f80a-4af7-90bf-2be049d010ee" containerName="revoke-keys"
Oct 31 20:02:22.335860 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.335850    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="78899e07-f80a-4af7-90bf-2be049d010ee" containerName="revoke-keys"
Oct 31 20:02:22.339635 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.339612    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:02:22.378959 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.378923    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk\" (UID: \"19e9ccd8-1762-45e1-aa02-b8478d5d59f6\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:02:22.379331 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.379163    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrv6d\" (UniqueName: \"kubernetes.io/projected/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-kube-api-access-qrv6d\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk\" (UID: \"19e9ccd8-1762-45e1-aa02-b8478d5d59f6\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:02:22.401910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.394664    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"]
Oct 31 20:02:22.426678 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.414583    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9902ffd9-dockercfg-29gzc\""
Oct 31 20:02:22.426678 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.414740    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9902ffd9\""
Oct 31 20:02:22.490677 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.489705    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk\" (UID: \"19e9ccd8-1762-45e1-aa02-b8478d5d59f6\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:02:22.500302 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.500186    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:22.503165 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.502025    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrv6d\" (UniqueName: \"kubernetes.io/projected/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-kube-api-access-qrv6d\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk\" (UID: \"19e9ccd8-1762-45e1-aa02-b8478d5d59f6\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:02:22.507033 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.507005    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk\" (UID: \"19e9ccd8-1762-45e1-aa02-b8478d5d59f6\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:02:22.547767 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.539306    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"]
Oct 31 20:02:22.563457 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.562077    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrv6d\" (UniqueName: \"kubernetes.io/projected/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-kube-api-access-qrv6d\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk\" (UID: \"19e9ccd8-1762-45e1-aa02-b8478d5d59f6\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:02:22.568073 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.567531    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:02:22.580194 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.580160    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"]
Oct 31 20:02:22.603377 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.603339    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwpwp\" (UniqueName: \"kubernetes.io/projected/3f769cfd-b755-4602-89ca-fd3522116a37-kube-api-access-dwpwp\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52\" (UID: \"3f769cfd-b755-4602-89ca-fd3522116a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:02:22.603594 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.603524    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f769cfd-b755-4602-89ca-fd3522116a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52\" (UID: \"3f769cfd-b755-4602-89ca-fd3522116a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:02:22.656109 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.656070    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:02:22.705673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.705159    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwpwp\" (UniqueName: \"kubernetes.io/projected/3f769cfd-b755-4602-89ca-fd3522116a37-kube-api-access-dwpwp\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52\" (UID: \"3f769cfd-b755-4602-89ca-fd3522116a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:02:22.705673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.705373    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f769cfd-b755-4602-89ca-fd3522116a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52\" (UID: \"3f769cfd-b755-4602-89ca-fd3522116a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:02:22.706364 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.706332    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f769cfd-b755-4602-89ca-fd3522116a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52\" (UID: \"3f769cfd-b755-4602-89ca-fd3522116a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:02:22.763803 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.763760    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwpwp\" (UniqueName: \"kubernetes.io/projected/3f769cfd-b755-4602-89ca-fd3522116a37-kube-api-access-dwpwp\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52\" (UID: \"3f769cfd-b755-4602-89ca-fd3522116a37\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:02:22.900279 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.900179    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:02:22.928127 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.928097    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"]
Oct 31 20:02:22.928403 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.928367    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy" containerID="cri-o://c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8" gracePeriod=30
Oct 31 20:02:22.940820 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.934755    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"]
Oct 31 20:02:22.940820 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.935011    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy" containerID="cri-o://c81a12347465d63d028daf1fca937429ebac49b0af147cd4e07b91051596aecc" gracePeriod=30
Oct 31 20:02:22.942709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.941830    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"]
Oct 31 20:02:22.942709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:22.942151    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy" containerID="cri-o://227d63be116789257900d498490fafcd26862c6e32a20bacaa9a59aa5084c2bc" gracePeriod=30
Oct 31 20:02:23.051048 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:02:23.050983    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e9ccd8_1762_45e1_aa02_b8478d5d59f6.slice/crio-0dd631d61ec460779d6aa5a6424f1f0c54e2e940de5c804a1b4926e76051a27b WatchSource:0}: Error finding container 0dd631d61ec460779d6aa5a6424f1f0c54e2e940de5c804a1b4926e76051a27b: Status 404 returned error can't find the container with id 0dd631d61ec460779d6aa5a6424f1f0c54e2e940de5c804a1b4926e76051a27b
Oct 31 20:02:23.053255 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:23.053209    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"]
Oct 31 20:02:23.325300 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:02:23.325262    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f769cfd_b755_4602_89ca_fd3522116a37.slice/crio-a741c66380d2f91de4d55c952d75505187d370e8a06cce79835301178d015c75 WatchSource:0}: Error finding container a741c66380d2f91de4d55c952d75505187d370e8a06cce79835301178d015c75: Status 404 returned error can't find the container with id a741c66380d2f91de4d55c952d75505187d370e8a06cce79835301178d015c75
Oct 31 20:02:23.328934 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:23.328796    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"]
Oct 31 20:02:23.331515 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:23.331478    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk" event={"ID":"19e9ccd8-1762-45e1-aa02-b8478d5d59f6","Type":"ContainerStarted","Data":"8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae"}
Oct 31 20:02:23.331983 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:23.331943    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:02:23.332044 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:23.331980    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk" event={"ID":"19e9ccd8-1762-45e1-aa02-b8478d5d59f6","Type":"ContainerStarted","Data":"0dd631d61ec460779d6aa5a6424f1f0c54e2e940de5c804a1b4926e76051a27b"}
Oct 31 20:02:23.446855 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:23.446785    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk" podStartSLOduration=1.446763077 podStartE2EDuration="1.446763077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:23.445737858 +0000 UTC m=+2761.975271930" watchObservedRunningTime="2026-10-31 20:02:23.446763077 +0000 UTC m=+2761.976297132"
Oct 31 20:02:24.120793 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:24.120742    5006 scope.go:122] "RemoveContainer" containerID="b3f6a87f33732df8b9c14345a287eeab828637984a626b8525c9e0efa2981b75"
Oct 31 20:02:24.352770 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:24.352709    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52" event={"ID":"3f769cfd-b755-4602-89ca-fd3522116a37","Type":"ContainerStarted","Data":"51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be"}
Oct 31 20:02:24.352972 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:24.352815    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52" event={"ID":"3f769cfd-b755-4602-89ca-fd3522116a37","Type":"ContainerStarted","Data":"a741c66380d2f91de4d55c952d75505187d370e8a06cce79835301178d015c75"}
Oct 31 20:02:24.352972 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:24.352843    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:02:24.481072 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:24.480967    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52" podStartSLOduration=2.480951911 podStartE2EDuration="2.480951911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:24.479015914 +0000 UTC m=+2763.008549967" watchObservedRunningTime="2026-10-31 20:02:24.480951911 +0000 UTC m=+2763.010485960"
Oct 31 20:02:25.537904 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:25.537868    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:27.066357 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:27.066286    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:02:27.066894 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:27.066363    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:02:27.235804 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:27.235766    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:27.448836 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:27.448783    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:28.431089 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.431029    5006 generic.go:356] "Generic (PLEG): container finished" podID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerID="227d63be116789257900d498490fafcd26862c6e32a20bacaa9a59aa5084c2bc" exitCode=0
Oct 31 20:02:28.431708 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.431234    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" event={"ID":"d61fd271-fe36-43f7-a548-0675c01ef58a","Type":"ContainerDied","Data":"227d63be116789257900d498490fafcd26862c6e32a20bacaa9a59aa5084c2bc"}
Oct 31 20:02:28.435972 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.435849    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 20:02:28.441460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.440307    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" event={"ID":"ed67b158-b0cf-4b1b-bb99-ba78733f2390","Type":"ContainerDied","Data":"c81a12347465d63d028daf1fca937429ebac49b0af147cd4e07b91051596aecc"}
Oct 31 20:02:28.441460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.440578    5006 generic.go:356] "Generic (PLEG): container finished" podID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerID="c81a12347465d63d028daf1fca937429ebac49b0af147cd4e07b91051596aecc" exitCode=0
Oct 31 20:02:28.443981 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.443632    5006 generic.go:356] "Generic (PLEG): container finished" podID="85b99532-ceaf-4202-b988-c4a88e738313" containerID="c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8" exitCode=0
Oct 31 20:02:28.444175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.444097    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" event={"ID":"85b99532-ceaf-4202-b988-c4a88e738313","Type":"ContainerDied","Data":"c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8"}
Oct 31 20:02:28.444254 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.444215    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg" event={"ID":"85b99532-ceaf-4202-b988-c4a88e738313","Type":"ContainerDied","Data":"62284eb902ad030e6abdc02b5ced709ef1b80315700e83b99367b6da56916683"}
Oct 31 20:02:28.444254 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.444250    5006 scope.go:122] "RemoveContainer" containerID="c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8"
Oct 31 20:02:28.446760 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.446688    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"
Oct 31 20:02:28.452360 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.452253    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 20:02:28.463197 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.463150    5006 scope.go:122] "RemoveContainer" containerID="c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8"
Oct 31 20:02:28.463604 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:02:28.463557    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8\": container with ID starting with c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8 not found: ID does not exist" containerID="c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8"
Oct 31 20:02:28.463822 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.463785    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8"} err="failed to get container status \"c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8\": rpc error: code = NotFound desc = could not find container \"c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8\": container with ID starting with c4addb7ab499586592da2ca91e081ddfec68bd08ab21a274d1ed984b64e62ca8 not found: ID does not exist"
Oct 31 20:02:28.539575 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.539535    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 20:02:28.559689 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.559611    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-kube-api-access-h8nbk\" (UniqueName: \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-kube-api-access-h8nbk\") pod \"d61fd271-fe36-43f7-a548-0675c01ef58a\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") "
Oct 31 20:02:28.559689 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.559683    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-data\") pod \"85b99532-ceaf-4202-b988-c4a88e738313\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") "
Oct 31 20:02:28.559948 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.559714    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-socket\") pod \"85b99532-ceaf-4202-b988-c4a88e738313\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") "
Oct 31 20:02:28.559948 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.559770    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-credential-socket\") pod \"85b99532-ceaf-4202-b988-c4a88e738313\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") "
Oct 31 20:02:28.559948 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.559820    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-token\" (UniqueName: \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-token\") pod \"d61fd271-fe36-43f7-a548-0675c01ef58a\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") "
Oct 31 20:02:28.559948 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.559867    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-kube-api-access-82dtc\" (UniqueName: \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-kube-api-access-82dtc\") pod \"85b99532-ceaf-4202-b988-c4a88e738313\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") "
Oct 31 20:02:28.559948 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.559946    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-istio-token\" (UniqueName: \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-istio-token\") pod \"85b99532-ceaf-4202-b988-c4a88e738313\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") "
Oct 31 20:02:28.560213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.559979    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b99532-ceaf-4202-b988-c4a88e738313-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b99532-ceaf-4202-b988-c4a88e738313-istiod-ca-cert\") pod \"85b99532-ceaf-4202-b988-c4a88e738313\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") "
Oct 31 20:02:28.560213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560060    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-socket\") pod \"d61fd271-fe36-43f7-a548-0675c01ef58a\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") "
Oct 31 20:02:28.560213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560117    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-credential-socket\") pod \"d61fd271-fe36-43f7-a548-0675c01ef58a\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") "
Oct 31 20:02:28.560213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560182    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-podinfo\") pod \"d61fd271-fe36-43f7-a548-0675c01ef58a\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") "
Oct 31 20:02:28.560213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560211    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-envoy\") pod \"d61fd271-fe36-43f7-a548-0675c01ef58a\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") "
Oct 31 20:02:28.560474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560227    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-credential-socket" pod "85b99532-ceaf-4202-b988-c4a88e738313" (UID: "85b99532-ceaf-4202-b988-c4a88e738313"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.560474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560254    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-data\") pod \"d61fd271-fe36-43f7-a548-0675c01ef58a\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") "
Oct 31 20:02:28.560474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560287    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85b99532-ceaf-4202-b988-c4a88e738313-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b99532-ceaf-4202-b988-c4a88e738313-istio-podinfo\") pod \"85b99532-ceaf-4202-b988-c4a88e738313\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") "
Oct 31 20:02:28.560474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560327    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61fd271-fe36-43f7-a548-0675c01ef58a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61fd271-fe36-43f7-a548-0675c01ef58a-istiod-ca-cert\") pod \"d61fd271-fe36-43f7-a548-0675c01ef58a\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") "
Oct 31 20:02:28.560474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560365    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-certs\") pod \"85b99532-ceaf-4202-b988-c4a88e738313\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") "
Oct 31 20:02:28.560474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560444    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-data" pod "85b99532-ceaf-4202-b988-c4a88e738313" (UID: "85b99532-ceaf-4202-b988-c4a88e738313"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.560875 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560685    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-data" pod "d61fd271-fe36-43f7-a548-0675c01ef58a" (UID: "d61fd271-fe36-43f7-a548-0675c01ef58a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.560875 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560696    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-socket" pod "85b99532-ceaf-4202-b988-c4a88e738313" (UID: "85b99532-ceaf-4202-b988-c4a88e738313"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.560875 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560737    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-certs" pod "85b99532-ceaf-4202-b988-c4a88e738313" (UID: "85b99532-ceaf-4202-b988-c4a88e738313"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.561122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.560909    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-socket" pod "d61fd271-fe36-43f7-a548-0675c01ef58a" (UID: "d61fd271-fe36-43f7-a548-0675c01ef58a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.561328 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.561216    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b99532-ceaf-4202-b988-c4a88e738313-istiod-ca-cert" pod "85b99532-ceaf-4202-b988-c4a88e738313" (UID: "85b99532-ceaf-4202-b988-c4a88e738313"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:28.561618 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.561592    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61fd271-fe36-43f7-a548-0675c01ef58a-istiod-ca-cert" pod "d61fd271-fe36-43f7-a548-0675c01ef58a" (UID: "d61fd271-fe36-43f7-a548-0675c01ef58a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:28.562025 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.561999    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-credential-socket" pod "d61fd271-fe36-43f7-a548-0675c01ef58a" (UID: "d61fd271-fe36-43f7-a548-0675c01ef58a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.562118 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.562070    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-certs\") pod \"d61fd271-fe36-43f7-a548-0675c01ef58a\" (UID: \"d61fd271-fe36-43f7-a548-0675c01ef58a\") "
Oct 31 20:02:28.562178 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.562115    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-envoy\") pod \"85b99532-ceaf-4202-b988-c4a88e738313\" (UID: \"85b99532-ceaf-4202-b988-c4a88e738313\") "
Oct 31 20:02:28.563055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563012    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.563055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563037    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b99532-ceaf-4202-b988-c4a88e738313-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.563055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563055    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.563259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563070    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.563259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563069    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-podinfo" pod "d61fd271-fe36-43f7-a548-0675c01ef58a" (UID: "d61fd271-fe36-43f7-a548-0675c01ef58a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:02:28.563259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563083    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.563259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563097    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61fd271-fe36-43f7-a548-0675c01ef58a-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.563259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563104    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-token" pod "d61fd271-fe36-43f7-a548-0675c01ef58a" (UID: "d61fd271-fe36-43f7-a548-0675c01ef58a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:28.563259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563110    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.563259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563146    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.563259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563165    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.563742 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563290    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-certs" pod "d61fd271-fe36-43f7-a548-0675c01ef58a" (UID: "d61fd271-fe36-43f7-a548-0675c01ef58a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.563742 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563726    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85b99532-ceaf-4202-b988-c4a88e738313-istio-podinfo" pod "85b99532-ceaf-4202-b988-c4a88e738313" (UID: "85b99532-ceaf-4202-b988-c4a88e738313"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:02:28.563853 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.563773    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-kube-api-access-82dtc" pod "85b99532-ceaf-4202-b988-c4a88e738313" (UID: "85b99532-ceaf-4202-b988-c4a88e738313"). InnerVolumeSpecName "kube-api-access-82dtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:28.564335 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.564312    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-kube-api-access-h8nbk" pod "d61fd271-fe36-43f7-a548-0675c01ef58a" (UID: "d61fd271-fe36-43f7-a548-0675c01ef58a"). InnerVolumeSpecName "kube-api-access-h8nbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:28.564808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.564782    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-istio-token" pod "85b99532-ceaf-4202-b988-c4a88e738313" (UID: "85b99532-ceaf-4202-b988-c4a88e738313"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:28.565019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.564980    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-envoy" pod "d61fd271-fe36-43f7-a548-0675c01ef58a" (UID: "d61fd271-fe36-43f7-a548-0675c01ef58a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.565358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.565338    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-envoy" pod "85b99532-ceaf-4202-b988-c4a88e738313" (UID: "85b99532-ceaf-4202-b988-c4a88e738313"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.664084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.663673    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-data\") pod \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") "
Oct 31 20:02:28.664084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.663742    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-envoy\") pod \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") "
Oct 31 20:02:28.664084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.663772    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-certs\") pod \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") "
Oct 31 20:02:28.664084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.663828    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-socket\") pod \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") "
Oct 31 20:02:28.664084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.663940    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istiod-ca-cert\") pod \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") "
Oct 31 20:02:28.664084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.663968    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-token\" (UniqueName: \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-token\") pod \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") "
Oct 31 20:02:28.664084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.664012    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-data" pod "ed67b158-b0cf-4b1b-bb99-ba78733f2390" (UID: "ed67b158-b0cf-4b1b-bb99-ba78733f2390"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.664084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.664027    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-kube-api-access-mm8bz\" (UniqueName: \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-kube-api-access-mm8bz\") pod \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") "
Oct 31 20:02:28.664685 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.664184    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-podinfo\") pod \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") "
Oct 31 20:02:28.664685 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.664277    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-credential-socket\") pod \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\" (UID: \"ed67b158-b0cf-4b1b-bb99-ba78733f2390\") "
Oct 31 20:02:28.664685 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.664543    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-socket" pod "ed67b158-b0cf-4b1b-bb99-ba78733f2390" (UID: "ed67b158-b0cf-4b1b-bb99-ba78733f2390"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.664855 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.664762    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-credential-socket" pod "ed67b158-b0cf-4b1b-bb99-ba78733f2390" (UID: "ed67b158-b0cf-4b1b-bb99-ba78733f2390"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.664983    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b99532-ceaf-4202-b988-c4a88e738313-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665015    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665030    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b99532-ceaf-4202-b988-c4a88e738313-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665032    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istiod-ca-cert" pod "ed67b158-b0cf-4b1b-bb99-ba78733f2390" (UID: "ed67b158-b0cf-4b1b-bb99-ba78733f2390"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665045    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8nbk\" (UniqueName: \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-kube-api-access-h8nbk\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665060    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665074    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82dtc\" (UniqueName: \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-kube-api-access-82dtc\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665089    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665103    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665116    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b99532-ceaf-4202-b988-c4a88e738313-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665132    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665145    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665159    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61fd271-fe36-43f7-a548-0675c01ef58a-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.665389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.665338    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-certs" pod "ed67b158-b0cf-4b1b-bb99-ba78733f2390" (UID: "ed67b158-b0cf-4b1b-bb99-ba78733f2390"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.667158 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.666548    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-envoy" pod "ed67b158-b0cf-4b1b-bb99-ba78733f2390" (UID: "ed67b158-b0cf-4b1b-bb99-ba78733f2390"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:28.667553 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.667527    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-kube-api-access-mm8bz" pod "ed67b158-b0cf-4b1b-bb99-ba78733f2390" (UID: "ed67b158-b0cf-4b1b-bb99-ba78733f2390"). InnerVolumeSpecName "kube-api-access-mm8bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:28.668048 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.668026    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-token" pod "ed67b158-b0cf-4b1b-bb99-ba78733f2390" (UID: "ed67b158-b0cf-4b1b-bb99-ba78733f2390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:28.668116 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.668098    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-podinfo" pod "ed67b158-b0cf-4b1b-bb99-ba78733f2390" (UID: "ed67b158-b0cf-4b1b-bb99-ba78733f2390"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:02:28.766322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.766282    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.766322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.766318    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.766322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.766328    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm8bz\" (UniqueName: \"kubernetes.io/projected/ed67b158-b0cf-4b1b-bb99-ba78733f2390-kube-api-access-mm8bz\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.766569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.766340    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.766569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.766350    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.766569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:28.766358    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed67b158-b0cf-4b1b-bb99-ba78733f2390-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:29.044177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.044121    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"]
Oct 31 20:02:29.049435 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.049396    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-psnsg"]
Oct 31 20:02:29.450158 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.450114    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"]
Oct 31 20:02:29.457916 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.457886    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"
Oct 31 20:02:29.458128 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.457890    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt" event={"ID":"d61fd271-fe36-43f7-a548-0675c01ef58a","Type":"ContainerDied","Data":"fc7a6330645b80a677e05acf35d0c3cadbf9a57245d968265e91beddd8a02332"}
Oct 31 20:02:29.458128 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.458041    5006 scope.go:122] "RemoveContainer" containerID="227d63be116789257900d498490fafcd26862c6e32a20bacaa9a59aa5084c2bc"
Oct 31 20:02:29.460969 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.460930    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr" event={"ID":"ed67b158-b0cf-4b1b-bb99-ba78733f2390","Type":"ContainerDied","Data":"c38e274699973a3dfc6ebcc55554811f6f4789a275d75c644fb7d405e320d94f"}
Oct 31 20:02:29.461254 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.461233    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"
Oct 31 20:02:29.473904 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.473882    5006 scope.go:122] "RemoveContainer" containerID="c81a12347465d63d028daf1fca937429ebac49b0af147cd4e07b91051596aecc"
Oct 31 20:02:29.599944 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.599897    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b99532-ceaf-4202-b988-c4a88e738313" path="/var/lib/kubelet/pods/85b99532-ceaf-4202-b988-c4a88e738313/volumes"
Oct 31 20:02:29.678938 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.678901    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"]
Oct 31 20:02:29.695703 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.695663    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-7wdrt"]
Oct 31 20:02:29.797052 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.795601    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:29.807546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.807488    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"]
Oct 31 20:02:29.810038 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:29.810009    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5dxr"]
Oct 31 20:02:30.011106 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:30.011066    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:30.386559 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:30.386494    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:31.600335 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:31.600287    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" path="/var/lib/kubelet/pods/d61fd271-fe36-43f7-a548-0675c01ef58a/volumes"
Oct 31 20:02:31.601150 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:31.601123    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" path="/var/lib/kubelet/pods/ed67b158-b0cf-4b1b-bb99-ba78733f2390/volumes"
Oct 31 20:02:33.288399 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:33.288360    5006 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 20:02:33.288931 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:33.288421    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 20:02:33.497679 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:33.497568    5006 generic.go:356] "Generic (PLEG): container finished" podID="db340a3a-dca2-42cd-84de-01422845fd59" containerID="522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc" exitCode=0
Oct 31 20:02:33.497679 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:33.497625    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" event={"ID":"db340a3a-dca2-42cd-84de-01422845fd59","Type":"ContainerDied","Data":"522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc"}
Oct 31 20:02:33.498118 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:33.498100    5006 scope.go:122] "RemoveContainer" containerID="522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc"
Oct 31 20:02:33.499786 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:33.499755    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:33.499857 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:33.499841    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:33.499913 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:33.499877    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:34.356094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.356066    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:02:34.450740 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.450692    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m" podUID="432c71ed-404c-4b6d-a521-0ed8ff30f8ea" containerName="payload-pre-processing" containerID="cri-o://8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd" gracePeriod=25
Oct 31 20:02:34.533365 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.533329    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" event={"ID":"db340a3a-dca2-42cd-84de-01422845fd59","Type":"ContainerStarted","Data":"716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a"}
Oct 31 20:02:34.583519 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.583482    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"]
Oct 31 20:02:34.584167 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.584150    5006 cpu_manager.go:414] "containerMap: removing container" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy"
Oct 31 20:02:34.584220 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.584168    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy"
Oct 31 20:02:34.584262 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.584224    5006 cpu_manager.go:414] "containerMap: removing container" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy"
Oct 31 20:02:34.584262 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.584232    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy"
Oct 31 20:02:34.584262 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.584249    5006 cpu_manager.go:414] "containerMap: removing container" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy"
Oct 31 20:02:34.584262 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.584254    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy"
Oct 31 20:02:34.584413 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.584382    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61fd271-fe36-43f7-a548-0675c01ef58a" containerName="istio-proxy"
Oct 31 20:02:34.584413 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.584395    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b99532-ceaf-4202-b988-c4a88e738313" containerName="istio-proxy"
Oct 31 20:02:34.584413 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.584402    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed67b158-b0cf-4b1b-bb99-ba78733f2390" containerName="istio-proxy"
Oct 31 20:02:34.588323 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.588272    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.632721 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.632626    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"]
Oct 31 20:02:34.637458 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.637417    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:34.639511 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.639485    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"]
Oct 31 20:02:34.640643 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.640615    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:02:34.640792 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.640772    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:02:34.641709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.641689    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:02:34.641928 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.641874    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:02:34.649426 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.649386    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsbng\" (UniqueName: \"kubernetes.io/projected/251545ff-dc9a-4999-9745-c4fec6d46289-kube-api-access-nsbng\") pod \"maas-api-revoke-sub-e8b32e68-zsc2k\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:34.649692 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.649585    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b32e68-zsc2k\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:34.649692 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.649683    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b32e68-zsc2k\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:34.649826 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.649717    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28f1d660-56mpl\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.649826 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.649763    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgbt\" (UniqueName: \"kubernetes.io/projected/35f9b171-42c4-4f58-a37b-9d2021007cd3-kube-api-access-cbgbt\") pod \"maas-api-revoke-sub-28f1d660-56mpl\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.649915 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.649863    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28f1d660-56mpl\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.690156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.690117    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"]
Oct 31 20:02:34.751274 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.751212    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b32e68-zsc2k\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:34.751512 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.751310    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b32e68-zsc2k\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:34.751512 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.751351    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28f1d660-56mpl\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.751512 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.751394    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgbt\" (UniqueName: \"kubernetes.io/projected/35f9b171-42c4-4f58-a37b-9d2021007cd3-kube-api-access-cbgbt\") pod \"maas-api-revoke-sub-28f1d660-56mpl\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.751512 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.751470    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28f1d660-56mpl\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.751512 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.751510    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsbng\" (UniqueName: \"kubernetes.io/projected/251545ff-dc9a-4999-9745-c4fec6d46289-kube-api-access-nsbng\") pod \"maas-api-revoke-sub-e8b32e68-zsc2k\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:34.752390 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.752355    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b32e68-zsc2k\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:34.752521 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.752460    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28f1d660-56mpl\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.754393 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.754368    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b32e68-zsc2k\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:34.754709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.754689    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28f1d660-56mpl\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.823625 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.823591    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsbng\" (UniqueName: \"kubernetes.io/projected/251545ff-dc9a-4999-9745-c4fec6d46289-kube-api-access-nsbng\") pod \"maas-api-revoke-sub-e8b32e68-zsc2k\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:34.823799 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.823680    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgbt\" (UniqueName: \"kubernetes.io/projected/35f9b171-42c4-4f58-a37b-9d2021007cd3-kube-api-access-cbgbt\") pod \"maas-api-revoke-sub-28f1d660-56mpl\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.929954 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.929856    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:34.958824 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:34.958793    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:35.326604 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:35.326570    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"]
Oct 31 20:02:35.329353 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:02:35.329322    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f9b171_42c4_4f58_a37b_9d2021007cd3.slice/crio-84c18a998a63f935f560759b9b3b3e5d5f6604d6dc0671e12f7ac5bed9d7580d WatchSource:0}: Error finding container 84c18a998a63f935f560759b9b3b3e5d5f6604d6dc0671e12f7ac5bed9d7580d: Status 404 returned error can't find the container with id 84c18a998a63f935f560759b9b3b3e5d5f6604d6dc0671e12f7ac5bed9d7580d
Oct 31 20:02:35.366783 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:35.366746    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:02:35.395543 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:02:35.395504    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:02:35.427489 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:35.427461    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"]
Oct 31 20:02:35.467128 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:35.467092    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:02:35.467621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:35.467599    5006 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 20:02:35.467708 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:35.467646    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 20:02:35.543505 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:35.543468    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:35.551760 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:35.551720    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl" event={"ID":"35f9b171-42c4-4f58-a37b-9d2021007cd3","Type":"ContainerStarted","Data":"84c18a998a63f935f560759b9b3b3e5d5f6604d6dc0671e12f7ac5bed9d7580d"}
Oct 31 20:02:36.467769 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:36.467730    5006 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 20:02:36.468163 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:36.467792    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 20:02:36.561721 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:36.561683    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl" event={"ID":"35f9b171-42c4-4f58-a37b-9d2021007cd3","Type":"ContainerStarted","Data":"c04b72ec16b2e920d391dca71c50f1a446af49cf5fc94fb129231da8830bb134"}
Oct 31 20:02:36.563587 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:36.563546    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k" event={"ID":"251545ff-dc9a-4999-9745-c4fec6d46289","Type":"ContainerStarted","Data":"07806075a9806701b0f0ecb9958ad78da8e244305ba8caf569ca321e8e5b465d"}
Oct 31 20:02:36.563804 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:36.563614    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k" event={"ID":"251545ff-dc9a-4999-9745-c4fec6d46289","Type":"ContainerStarted","Data":"a243159d84cc1ec71f4497b68b33c01ee9eae7bd22fb79abb2c14d5b7002771c"}
Oct 31 20:02:36.726621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:36.726491    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl" podStartSLOduration=2.726468204 podStartE2EDuration="2.726468204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:36.722596971 +0000 UTC m=+2775.252131050" watchObservedRunningTime="2026-10-31 20:02:36.726468204 +0000 UTC m=+2775.256002259"
Oct 31 20:02:36.840815 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:36.840754    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k" podStartSLOduration=2.840738592 podStartE2EDuration="2.840738592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:36.836750381 +0000 UTC m=+2775.366284431" watchObservedRunningTime="2026-10-31 20:02:36.840738592 +0000 UTC m=+2775.370272644"
Oct 31 20:02:37.281700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.281633    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" containerID="cri-o://23c102c5d5cdf1c335681ca3dfb545dccb9a6e15d2335e67f6ffdeb1800dc789" gracePeriod=45
Oct 31 20:02:37.294208 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.294162    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" containerID="cri-o://3994e23e1a948b527e2966719a25dab0a5da12a34cd30b819006fbaf68dbd094" gracePeriod=45
Oct 31 20:02:37.304365 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.303140    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused"
Oct 31 20:02:37.325099 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.325056    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": EOF"
Oct 31 20:02:37.467562 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.467515    5006 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 20:02:37.467811 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.467585    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 20:02:37.583611 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.583575    5006 generic.go:356] "Generic (PLEG): container finished" podID="251545ff-dc9a-4999-9745-c4fec6d46289" containerID="07806075a9806701b0f0ecb9958ad78da8e244305ba8caf569ca321e8e5b465d" exitCode=0
Oct 31 20:02:37.583850 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.583748    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k" event={"ID":"251545ff-dc9a-4999-9745-c4fec6d46289","Type":"ContainerDied","Data":"07806075a9806701b0f0ecb9958ad78da8e244305ba8caf569ca321e8e5b465d"}
Oct 31 20:02:37.586124 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.586091    5006 generic.go:356] "Generic (PLEG): container finished" podID="da2d3282-0a38-4317-a213-9bba58d67932" containerID="23c102c5d5cdf1c335681ca3dfb545dccb9a6e15d2335e67f6ffdeb1800dc789" exitCode=0
Oct 31 20:02:37.586280 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.586181    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" event={"ID":"da2d3282-0a38-4317-a213-9bba58d67932","Type":"ContainerDied","Data":"23c102c5d5cdf1c335681ca3dfb545dccb9a6e15d2335e67f6ffdeb1800dc789"}
Oct 31 20:02:37.588402 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.588370    5006 generic.go:356] "Generic (PLEG): container finished" podID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerID="3994e23e1a948b527e2966719a25dab0a5da12a34cd30b819006fbaf68dbd094" exitCode=0
Oct 31 20:02:37.588520 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.588406    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" event={"ID":"2e0cc5cc-1d03-4aab-970d-54b15f432bd4","Type":"ContainerDied","Data":"3994e23e1a948b527e2966719a25dab0a5da12a34cd30b819006fbaf68dbd094"}
Oct 31 20:02:37.591048 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.591024    5006 generic.go:356] "Generic (PLEG): container finished" podID="35f9b171-42c4-4f58-a37b-9d2021007cd3" containerID="c04b72ec16b2e920d391dca71c50f1a446af49cf5fc94fb129231da8830bb134" exitCode=0
Oct 31 20:02:37.591187 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.591057    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl" event={"ID":"35f9b171-42c4-4f58-a37b-9d2021007cd3","Type":"ContainerDied","Data":"c04b72ec16b2e920d391dca71c50f1a446af49cf5fc94fb129231da8830bb134"}
Oct 31 20:02:37.651535 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.651503    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 20:02:37.673906 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.673879    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 20:02:37.732981 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.732944    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2d3282-0a38-4317-a213-9bba58d67932-kube-api-access-tt9sc\" (UniqueName: \"kubernetes.io/projected/da2d3282-0a38-4317-a213-9bba58d67932-kube-api-access-tt9sc\") pod \"da2d3282-0a38-4317-a213-9bba58d67932\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") "
Oct 31 20:02:37.733206 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733006    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-model-cache\") pod \"da2d3282-0a38-4317-a213-9bba58d67932\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") "
Oct 31 20:02:37.733206 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733048    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-kube-api-access-jbsm8\" (UniqueName: \"kubernetes.io/projected/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-kube-api-access-jbsm8\") pod \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") "
Oct 31 20:02:37.733206 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733090    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-dshm\") pod \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") "
Oct 31 20:02:37.733206 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733195    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da2d3282-0a38-4317-a213-9bba58d67932-tls-certs\" (UniqueName: \"kubernetes.io/secret/da2d3282-0a38-4317-a213-9bba58d67932-tls-certs\") pod \"da2d3282-0a38-4317-a213-9bba58d67932\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") "
Oct 31 20:02:37.733436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733225    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-dshm\") pod \"da2d3282-0a38-4317-a213-9bba58d67932\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") "
Oct 31 20:02:37.733436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733292    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-tmp-dir\") pod \"da2d3282-0a38-4317-a213-9bba58d67932\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") "
Oct 31 20:02:37.733436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733337    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-model-cache" pod "da2d3282-0a38-4317-a213-9bba58d67932" (UID: "da2d3282-0a38-4317-a213-9bba58d67932"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:37.733436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733364    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tls-certs\") pod \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") "
Oct 31 20:02:37.733436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733408    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-model-cache\") pod \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") "
Oct 31 20:02:37.733627 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733442    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tmp-dir\") pod \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") "
Oct 31 20:02:37.733627 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733489    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-home\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-home\") pod \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\" (UID: \"2e0cc5cc-1d03-4aab-970d-54b15f432bd4\") "
Oct 31 20:02:37.733627 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733534    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-home\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-home\") pod \"da2d3282-0a38-4317-a213-9bba58d67932\" (UID: \"da2d3282-0a38-4317-a213-9bba58d67932\") "
Oct 31 20:02:37.734024 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.733990    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-model-cache" pod "2e0cc5cc-1d03-4aab-970d-54b15f432bd4" (UID: "2e0cc5cc-1d03-4aab-970d-54b15f432bd4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:37.734191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.734164    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-tmp-dir" pod "da2d3282-0a38-4317-a213-9bba58d67932" (UID: "da2d3282-0a38-4317-a213-9bba58d67932"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:37.734531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.734504    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tmp-dir" pod "2e0cc5cc-1d03-4aab-970d-54b15f432bd4" (UID: "2e0cc5cc-1d03-4aab-970d-54b15f432bd4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:37.734718 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.734641    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.734793 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.734726    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-home" pod "2e0cc5cc-1d03-4aab-970d-54b15f432bd4" (UID: "2e0cc5cc-1d03-4aab-970d-54b15f432bd4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:37.734923 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.734853    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-home" pod "da2d3282-0a38-4317-a213-9bba58d67932" (UID: "da2d3282-0a38-4317-a213-9bba58d67932"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:37.735093 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.735058    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.736419 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.736375    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-dshm" pod "da2d3282-0a38-4317-a213-9bba58d67932" (UID: "da2d3282-0a38-4317-a213-9bba58d67932"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:37.737307 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.737274    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da2d3282-0a38-4317-a213-9bba58d67932-tls-certs" pod "da2d3282-0a38-4317-a213-9bba58d67932" (UID: "da2d3282-0a38-4317-a213-9bba58d67932"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:37.737554 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.737530    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tls-certs" pod "2e0cc5cc-1d03-4aab-970d-54b15f432bd4" (UID: "2e0cc5cc-1d03-4aab-970d-54b15f432bd4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:37.737724 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.737551    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-dshm" pod "2e0cc5cc-1d03-4aab-970d-54b15f432bd4" (UID: "2e0cc5cc-1d03-4aab-970d-54b15f432bd4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:37.737724 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.737593    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2d3282-0a38-4317-a213-9bba58d67932-kube-api-access-tt9sc" pod "da2d3282-0a38-4317-a213-9bba58d67932" (UID: "da2d3282-0a38-4317-a213-9bba58d67932"). InnerVolumeSpecName "kube-api-access-tt9sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:37.738277 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.738256    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-kube-api-access-jbsm8" pod "2e0cc5cc-1d03-4aab-970d-54b15f432bd4" (UID: "2e0cc5cc-1d03-4aab-970d-54b15f432bd4"). InnerVolumeSpecName "kube-api-access-jbsm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:37.836393 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.836361    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.836393 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.836392    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.836393 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.836401    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.836737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.836410    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.836737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.836420    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt9sc\" (UniqueName: \"kubernetes.io/projected/da2d3282-0a38-4317-a213-9bba58d67932-kube-api-access-tt9sc\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.836737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.836430    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbsm8\" (UniqueName: \"kubernetes.io/projected/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-kube-api-access-jbsm8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.836737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.836440    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e0cc5cc-1d03-4aab-970d-54b15f432bd4-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.836737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.836449    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da2d3282-0a38-4317-a213-9bba58d67932-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.836737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.836457    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.836737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.836465    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da2d3282-0a38-4317-a213-9bba58d67932-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.932953 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.932564    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"]
Oct 31 20:02:37.933221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:37.933043    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy" containerID="cri-o://2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981" gracePeriod=30
Oct 31 20:02:38.467632 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.467591    5006 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 20:02:38.467851 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.467704    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 20:02:38.625100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.625055    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"
Oct 31 20:02:38.625283 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.625066    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55" event={"ID":"da2d3282-0a38-4317-a213-9bba58d67932","Type":"ContainerDied","Data":"223a1fb64193fd9cced2e55b333f9e1677ed94a61f930d1bb013168011b022c2"}
Oct 31 20:02:38.625283 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.625199    5006 scope.go:122] "RemoveContainer" containerID="23c102c5d5cdf1c335681ca3dfb545dccb9a6e15d2335e67f6ffdeb1800dc789"
Oct 31 20:02:38.627194 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.627166    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"
Oct 31 20:02:38.627316 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.627286    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b" event={"ID":"2e0cc5cc-1d03-4aab-970d-54b15f432bd4","Type":"ContainerDied","Data":"20994f2f6a1a5c342f568766247e8b354e96c75537f6dcc971ecb014fcf6f4d5"}
Oct 31 20:02:38.640425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.640291    5006 scope.go:122] "RemoveContainer" containerID="3994e23e1a948b527e2966719a25dab0a5da12a34cd30b819006fbaf68dbd094"
Oct 31 20:02:38.668771 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:02:38.668728    5006 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e0cc5cc_1d03_4aab_970d_54b15f432bd4.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 20:02:38.815690 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.814424    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"]
Oct 31 20:02:38.815690 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.814495    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/premium-w4-7cfafd-kserve-85fb74694f-grr55"]
Oct 31 20:02:38.929731 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.929698    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"]
Oct 31 20:02:38.933021 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.932989    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/simulator-w4-7cfafd-kserve-8569d84d5b-qbt2b"]
Oct 31 20:02:38.938391 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.938367    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:38.941138 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.941116    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:38.946088 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:38.946056    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m" podUID="432c71ed-404c-4b6d-a521-0ed8ff30f8ea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:02:39.074336 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.074247    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-xks-ca\") pod \"35f9b171-42c4-4f58-a37b-9d2021007cd3\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") "
Oct 31 20:02:39.074336 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.074323    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f9b171-42c4-4f58-a37b-9d2021007cd3-kube-api-access-cbgbt\" (UniqueName: \"kubernetes.io/projected/35f9b171-42c4-4f58-a37b-9d2021007cd3-kube-api-access-cbgbt\") pod \"35f9b171-42c4-4f58-a37b-9d2021007cd3\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") "
Oct 31 20:02:39.074586 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.074437    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251545ff-dc9a-4999-9745-c4fec6d46289-kube-api-access-nsbng\" (UniqueName: \"kubernetes.io/projected/251545ff-dc9a-4999-9745-c4fec6d46289-kube-api-access-nsbng\") pod \"251545ff-dc9a-4999-9745-c4fec6d46289\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") "
Oct 31 20:02:39.074586 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.074463    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-openshift-ca\") pod \"35f9b171-42c4-4f58-a37b-9d2021007cd3\" (UID: \"35f9b171-42c4-4f58-a37b-9d2021007cd3\") "
Oct 31 20:02:39.074586 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.074527    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-xks-ca\") pod \"251545ff-dc9a-4999-9745-c4fec6d46289\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") "
Oct 31 20:02:39.074805 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.074616    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-openshift-ca\") pod \"251545ff-dc9a-4999-9745-c4fec6d46289\" (UID: \"251545ff-dc9a-4999-9745-c4fec6d46289\") "
Oct 31 20:02:39.075639 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.074928    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-openshift-ca" pod "35f9b171-42c4-4f58-a37b-9d2021007cd3" (UID: "35f9b171-42c4-4f58-a37b-9d2021007cd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:39.075639 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.075183    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-openshift-ca" pod "251545ff-dc9a-4999-9745-c4fec6d46289" (UID: "251545ff-dc9a-4999-9745-c4fec6d46289"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:39.075639 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.075280    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:39.075639 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.075305    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:39.077129 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.077105    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f9b171-42c4-4f58-a37b-9d2021007cd3-kube-api-access-cbgbt" pod "35f9b171-42c4-4f58-a37b-9d2021007cd3" (UID: "35f9b171-42c4-4f58-a37b-9d2021007cd3"). InnerVolumeSpecName "kube-api-access-cbgbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:39.077257 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.077234    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-xks-ca" pod "251545ff-dc9a-4999-9745-c4fec6d46289" (UID: "251545ff-dc9a-4999-9745-c4fec6d46289"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:39.080191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.080162    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251545ff-dc9a-4999-9745-c4fec6d46289-kube-api-access-nsbng" pod "251545ff-dc9a-4999-9745-c4fec6d46289" (UID: "251545ff-dc9a-4999-9745-c4fec6d46289"). InnerVolumeSpecName "kube-api-access-nsbng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:39.080280 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.080191    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-xks-ca" pod "35f9b171-42c4-4f58-a37b-9d2021007cd3" (UID: "35f9b171-42c4-4f58-a37b-9d2021007cd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:39.177312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.177267    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251545ff-dc9a-4999-9745-c4fec6d46289-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:39.177312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.177302    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f9b171-42c4-4f58-a37b-9d2021007cd3-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:39.177312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.177317    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbgbt\" (UniqueName: \"kubernetes.io/projected/35f9b171-42c4-4f58-a37b-9d2021007cd3-kube-api-access-cbgbt\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:39.177699 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.177332    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsbng\" (UniqueName: \"kubernetes.io/projected/251545ff-dc9a-4999-9745-c4fec6d46289-kube-api-access-nsbng\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:39.467369 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.467324    5006 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 31 20:02:39.467588 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.467385    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 31 20:02:39.624018 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.623980    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" path="/var/lib/kubelet/pods/2e0cc5cc-1d03-4aab-970d-54b15f432bd4/volumes"
Oct 31 20:02:39.624578 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.624552    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2d3282-0a38-4317-a213-9bba58d67932" path="/var/lib/kubelet/pods/da2d3282-0a38-4317-a213-9bba58d67932/volumes"
Oct 31 20:02:39.635728 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.635688    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k" event={"ID":"251545ff-dc9a-4999-9745-c4fec6d46289","Type":"ContainerDied","Data":"a243159d84cc1ec71f4497b68b33c01ee9eae7bd22fb79abb2c14d5b7002771c"}
Oct 31 20:02:39.635728 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.635726    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"
Oct 31 20:02:39.635973 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.635750    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a243159d84cc1ec71f4497b68b33c01ee9eae7bd22fb79abb2c14d5b7002771c"
Oct 31 20:02:39.639799 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.639773    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl" event={"ID":"35f9b171-42c4-4f58-a37b-9d2021007cd3","Type":"ContainerDied","Data":"84c18a998a63f935f560759b9b3b3e5d5f6604d6dc0671e12f7ac5bed9d7580d"}
Oct 31 20:02:39.639882 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.639811    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"
Oct 31 20:02:39.639882 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:39.639819    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84c18a998a63f935f560759b9b3b3e5d5f6604d6dc0671e12f7ac5bed9d7580d"
Oct 31 20:02:40.473478 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:40.473443    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:02:40.473936 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:40.473913    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:02:40.474747 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:40.474727    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:02:41.426877 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:41.426837    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:41.695121 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:41.695023    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:41.880993 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:41.880958    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"]
Oct 31 20:02:41.909902 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:41.909857    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:41.925470 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:41.925434    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dfe658-2slfb"]
Oct 31 20:02:42.377857 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:42.377822    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:43.203003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.202968    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 20:02:43.325751 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.325632    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-credential-socket\") pod \"4299d2a8-888c-44c5-89d2-98275371afc7\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") "
Oct 31 20:02:43.325944 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.325814    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4299d2a8-888c-44c5-89d2-98275371afc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4299d2a8-888c-44c5-89d2-98275371afc7-istiod-ca-cert\") pod \"4299d2a8-888c-44c5-89d2-98275371afc7\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") "
Oct 31 20:02:43.325944 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.325853    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-certs\") pod \"4299d2a8-888c-44c5-89d2-98275371afc7\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") "
Oct 31 20:02:43.325944 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.325897    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-socket\") pod \"4299d2a8-888c-44c5-89d2-98275371afc7\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") "
Oct 31 20:02:43.326116 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.325948    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-data\") pod \"4299d2a8-888c-44c5-89d2-98275371afc7\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") "
Oct 31 20:02:43.326116 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.325961    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-credential-socket" pod "4299d2a8-888c-44c5-89d2-98275371afc7" (UID: "4299d2a8-888c-44c5-89d2-98275371afc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:43.326116 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326062    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4299d2a8-888c-44c5-89d2-98275371afc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4299d2a8-888c-44c5-89d2-98275371afc7-istio-podinfo\") pod \"4299d2a8-888c-44c5-89d2-98275371afc7\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") "
Oct 31 20:02:43.326116 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326097    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-istio-token\" (UniqueName: \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-istio-token\") pod \"4299d2a8-888c-44c5-89d2-98275371afc7\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") "
Oct 31 20:02:43.326394 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326165    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-envoy\") pod \"4299d2a8-888c-44c5-89d2-98275371afc7\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") "
Oct 31 20:02:43.326394 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326147    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-certs" pod "4299d2a8-888c-44c5-89d2-98275371afc7" (UID: "4299d2a8-888c-44c5-89d2-98275371afc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:43.326394 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326195    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-kube-api-access-bzdhs\" (UniqueName: \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-kube-api-access-bzdhs\") pod \"4299d2a8-888c-44c5-89d2-98275371afc7\" (UID: \"4299d2a8-888c-44c5-89d2-98275371afc7\") "
Oct 31 20:02:43.326394 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326224    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-socket" pod "4299d2a8-888c-44c5-89d2-98275371afc7" (UID: "4299d2a8-888c-44c5-89d2-98275371afc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:43.326394 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326280    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-data" pod "4299d2a8-888c-44c5-89d2-98275371afc7" (UID: "4299d2a8-888c-44c5-89d2-98275371afc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:43.326394 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326232    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4299d2a8-888c-44c5-89d2-98275371afc7-istiod-ca-cert" pod "4299d2a8-888c-44c5-89d2-98275371afc7" (UID: "4299d2a8-888c-44c5-89d2-98275371afc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:43.326771 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326748    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:43.326840 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326778    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4299d2a8-888c-44c5-89d2-98275371afc7-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:43.326840 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326792    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:43.326840 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326807    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:43.326840 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.326820    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:43.328301 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.328268    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4299d2a8-888c-44c5-89d2-98275371afc7-istio-podinfo" pod "4299d2a8-888c-44c5-89d2-98275371afc7" (UID: "4299d2a8-888c-44c5-89d2-98275371afc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:02:43.328457 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.328435    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-istio-token" pod "4299d2a8-888c-44c5-89d2-98275371afc7" (UID: "4299d2a8-888c-44c5-89d2-98275371afc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:43.328696 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.328668    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-envoy" pod "4299d2a8-888c-44c5-89d2-98275371afc7" (UID: "4299d2a8-888c-44c5-89d2-98275371afc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:43.328780 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.328727    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-kube-api-access-bzdhs" pod "4299d2a8-888c-44c5-89d2-98275371afc7" (UID: "4299d2a8-888c-44c5-89d2-98275371afc7"). InnerVolumeSpecName "kube-api-access-bzdhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:43.427977 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.427932    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4299d2a8-888c-44c5-89d2-98275371afc7-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:43.427977 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.427973    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:43.427977 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.427983    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4299d2a8-888c-44c5-89d2-98275371afc7-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:43.428247 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.427993    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzdhs\" (UniqueName: \"kubernetes.io/projected/4299d2a8-888c-44c5-89d2-98275371afc7-kube-api-access-bzdhs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:43.600027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.599993    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26abcc6-a5f1-48b8-b829-6a610e7307be" path="/var/lib/kubelet/pods/c26abcc6-a5f1-48b8-b829-6a610e7307be/volumes"
Oct 31 20:02:43.675808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.675770    5006 generic.go:356] "Generic (PLEG): container finished" podID="4299d2a8-888c-44c5-89d2-98275371afc7" containerID="2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981" exitCode=0
Oct 31 20:02:43.675963 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.675829    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"
Oct 31 20:02:43.675963 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.675862    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" event={"ID":"4299d2a8-888c-44c5-89d2-98275371afc7","Type":"ContainerDied","Data":"2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981"}
Oct 31 20:02:43.675963 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.675918    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx" event={"ID":"4299d2a8-888c-44c5-89d2-98275371afc7","Type":"ContainerDied","Data":"52468f683cba0db4ff547d22b099ada2737fbd343af825b969c88c1d1d6133bb"}
Oct 31 20:02:43.675963 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.675942    5006 scope.go:122] "RemoveContainer" containerID="2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981"
Oct 31 20:02:43.687117 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.687094    5006 scope.go:122] "RemoveContainer" containerID="2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981"
Oct 31 20:02:43.687431 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:02:43.687403    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981\": container with ID starting with 2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981 not found: ID does not exist" containerID="2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981"
Oct 31 20:02:43.687535 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.687440    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981"} err="failed to get container status \"2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981\": rpc error: code = NotFound desc = could not find container \"2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981\": container with ID starting with 2a108d9999c8dd11fc502be0f4a0be6f8da3fdea144527211773470590a89981 not found: ID does not exist"
Oct 31 20:02:43.844073 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.844022    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"]
Oct 31 20:02:43.851483 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:43.851381    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jrqmx"]
Oct 31 20:02:45.599029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:45.598992    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" path="/var/lib/kubelet/pods/4299d2a8-888c-44c5-89d2-98275371afc7/volumes"
Oct 31 20:02:46.292329 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:46.292282    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:47.999240 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:47.999205    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:48.284194 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:48.284097    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:48.471910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:48.471873    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"]
Oct 31 20:02:48.474872 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:48.474843    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ddb5a8-5f5p7"]
Oct 31 20:02:48.591070 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:48.591042    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:48.811041 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:48.811004    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:48.947135 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:48.946977    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m" podUID="432c71ed-404c-4b6d-a521-0ed8ff30f8ea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:02:49.039537 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:49.039495    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:49.260018 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:49.259934    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:49.599423 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:49.599389    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4add7c-97cb-4e50-8619-7809d82ff0f7" path="/var/lib/kubelet/pods/8c4add7c-97cb-4e50-8619-7809d82ff0f7/volumes"
Oct 31 20:02:51.211863 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.211785    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:51.631284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631245    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"]
Oct 31 20:02:51.631858 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631842    5006 cpu_manager.go:414] "containerMap: removing container" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy"
Oct 31 20:02:51.631910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631861    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy"
Oct 31 20:02:51.631910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631884    5006 cpu_manager.go:414] "containerMap: removing container" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main"
Oct 31 20:02:51.631910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631889    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main"
Oct 31 20:02:51.631910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631898    5006 cpu_manager.go:414] "containerMap: removing container" podUID="35f9b171-42c4-4f58-a37b-9d2021007cd3" containerName="revoke-keys"
Oct 31 20:02:51.631910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631903    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f9b171-42c4-4f58-a37b-9d2021007cd3" containerName="revoke-keys"
Oct 31 20:02:51.632085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631918    5006 cpu_manager.go:414] "containerMap: removing container" podUID="251545ff-dc9a-4999-9745-c4fec6d46289" containerName="revoke-keys"
Oct 31 20:02:51.632085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631924    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251545ff-dc9a-4999-9745-c4fec6d46289" containerName="revoke-keys"
Oct 31 20:02:51.632085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631932    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main"
Oct 31 20:02:51.632085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.631937    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main"
Oct 31 20:02:51.632085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.632026    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="251545ff-dc9a-4999-9745-c4fec6d46289" containerName="revoke-keys"
Oct 31 20:02:51.632085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.632035    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0cc5cc-1d03-4aab-970d-54b15f432bd4" containerName="main"
Oct 31 20:02:51.632085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.632047    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4299d2a8-888c-44c5-89d2-98275371afc7" containerName="istio-proxy"
Oct 31 20:02:51.632085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.632054    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2d3282-0a38-4317-a213-9bba58d67932" containerName="main"
Oct 31 20:02:51.632085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.632059    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f9b171-42c4-4f58-a37b-9d2021007cd3" containerName="revoke-keys"
Oct 31 20:02:51.636736 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.636709    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.690062 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.690027    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a89a58-openshift-default-dockercfg-7t2vh\""
Oct 31 20:02:51.754969 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.754914    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-envoy\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.755275 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.755252    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-credential-socket\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.755416 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.755401    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l8ft\" (UniqueName: \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-kube-api-access-4l8ft\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.755576 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.755560    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-data\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.755732 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.755714    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-token\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.755855 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.755839    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d8c84b-aefa-4e42-b121-33efa295c4b2-istiod-ca-cert\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.756032 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.756018    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-podinfo\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.756233 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.756220    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-socket\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.756338 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.756323    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-certs\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.795358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.795325    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"]
Oct 31 20:02:51.803703 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.800440    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:51.857819 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.857773    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-socket\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.857819 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.857817    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-certs\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.858094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.857848    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-envoy\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.858094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.857908    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-credential-socket\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.858094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.857942    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l8ft\" (UniqueName: \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-kube-api-access-4l8ft\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.858094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.858013    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-data\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.858094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.858063    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-token\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.858094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.858094    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d8c84b-aefa-4e42-b121-33efa295c4b2-istiod-ca-cert\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.858421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.858177    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-podinfo\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.858421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.858276    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-socket\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.858421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.858379    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-certs\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.859048 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.859020    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-data\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.859175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.859020    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-credential-socket\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.859905 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.859883    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d8c84b-aefa-4e42-b121-33efa295c4b2-istiod-ca-cert\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.862098 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.861045    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-envoy\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.863947 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.863907    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-podinfo\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.916448 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.916362    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-token\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.916599 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.916493    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l8ft\" (UniqueName: \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-kube-api-access-4l8ft\") pod \"e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") " pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:51.959457 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:51.959390    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:52.087799 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:52.087761    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:52.406222 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:02:52.406170    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d8c84b_aefa_4e42_b121_33efa295c4b2.slice/crio-59e7dce5a2e06e5821ba37dfefe6ccc80c3ec6be21bfe5b319fd03046e451643 WatchSource:0}: Error finding container 59e7dce5a2e06e5821ba37dfefe6ccc80c3ec6be21bfe5b319fd03046e451643: Status 404 returned error can't find the container with id 59e7dce5a2e06e5821ba37dfefe6ccc80c3ec6be21bfe5b319fd03046e451643
Oct 31 20:02:52.409325 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:52.409287    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:52.409439 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:52.409358    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:52.409439 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:52.409385    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:52.409755 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:52.409727    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"]
Oct 31 20:02:52.809075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:52.808984    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" event={"ID":"24d8c84b-aefa-4e42-b121-33efa295c4b2","Type":"ContainerStarted","Data":"f84eea95a62cbd4f5bfed203f94c14fe9baaef7fe72aa61f743c8a28b2224982"}
Oct 31 20:02:52.809075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:52.809042    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" event={"ID":"24d8c84b-aefa-4e42-b121-33efa295c4b2","Type":"ContainerStarted","Data":"59e7dce5a2e06e5821ba37dfefe6ccc80c3ec6be21bfe5b319fd03046e451643"}
Oct 31 20:02:52.928059 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:52.927990    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" podStartSLOduration=1.9279708690000001 podStartE2EDuration="1.927970869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:52.926227695 +0000 UTC m=+2791.455761758" watchObservedRunningTime="2026-10-31 20:02:52.927970869 +0000 UTC m=+2791.457504925"
Oct 31 20:02:52.960386 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:52.960350    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:53.663935 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:53.663893    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:53.965475 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:53.965398    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:54.734878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.734845    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 20:02:54.805008 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.804969    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-kube-api-access-2ftcb\" (UniqueName: \"kubernetes.io/projected/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-kube-api-access-2ftcb\") pod \"432c71ed-404c-4b6d-a521-0ed8ff30f8ea\" (UID: \"432c71ed-404c-4b6d-a521-0ed8ff30f8ea\") "
Oct 31 20:02:54.805228 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.805069    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-plugins-config-volume\") pod \"432c71ed-404c-4b6d-a521-0ed8ff30f8ea\" (UID: \"432c71ed-404c-4b6d-a521-0ed8ff30f8ea\") "
Oct 31 20:02:54.805525 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.805500    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-plugins-config-volume" pod "432c71ed-404c-4b6d-a521-0ed8ff30f8ea" (UID: "432c71ed-404c-4b6d-a521-0ed8ff30f8ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:54.805692 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.805642    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:54.807336 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.807306    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-kube-api-access-2ftcb" pod "432c71ed-404c-4b6d-a521-0ed8ff30f8ea" (UID: "432c71ed-404c-4b6d-a521-0ed8ff30f8ea"). InnerVolumeSpecName "kube-api-access-2ftcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:54.824760 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.824726    5006 generic.go:356] "Generic (PLEG): container finished" podID="432c71ed-404c-4b6d-a521-0ed8ff30f8ea" containerID="8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd" exitCode=0
Oct 31 20:02:54.824958 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.824777    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"
Oct 31 20:02:54.824958 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.824810    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m" event={"ID":"432c71ed-404c-4b6d-a521-0ed8ff30f8ea","Type":"ContainerDied","Data":"8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd"}
Oct 31 20:02:54.824958 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.824864    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m" event={"ID":"432c71ed-404c-4b6d-a521-0ed8ff30f8ea","Type":"ContainerDied","Data":"98c78cd75faf30cce2418c9c77161ac5efc5095b7de3ae6a1ce0aac83e9c0be9"}
Oct 31 20:02:54.824958 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.824886    5006 scope.go:122] "RemoveContainer" containerID="8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd"
Oct 31 20:02:54.825488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.825465    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:54.826725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.826543    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:02:54.836738 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.836711    5006 scope.go:122] "RemoveContainer" containerID="8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd"
Oct 31 20:02:54.837058 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:02:54.837036    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd\": container with ID starting with 8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd not found: ID does not exist" containerID="8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd"
Oct 31 20:02:54.837122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.837069    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd"} err="failed to get container status \"8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd\": rpc error: code = NotFound desc = could not find container \"8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd\": container with ID starting with 8887f624fb31d40d72d3c7d3902b1979b842e4eaaedf0f9d45a03482c65f65cd not found: ID does not exist"
Oct 31 20:02:54.907427 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:54.907390    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ftcb\" (UniqueName: \"kubernetes.io/projected/432c71ed-404c-4b6d-a521-0ed8ff30f8ea-kube-api-access-2ftcb\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:55.108011 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:55.107975    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"]
Oct 31 20:02:55.121114 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:55.121079    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7cfafd-795b67bc67-2hw9m"]
Oct 31 20:02:55.416152 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:55.416058    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:55.601109 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:55.601066    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432c71ed-404c-4b6d-a521-0ed8ff30f8ea" path="/var/lib/kubelet/pods/432c71ed-404c-4b6d-a521-0ed8ff30f8ea/volumes"
Oct 31 20:02:55.642191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:55.642158    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:55.861957 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:55.861914    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:56.098596 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:56.098559    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:56.379989 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:56.379958    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:56.790157 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:56.790070    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:57.169333 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:57.169301    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:58.895778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:58.895708    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:59.109517 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.109471    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:59.324892 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.324854    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:59.523779 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.523639    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"]
Oct 31 20:02:59.524423 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.524397    5006 cpu_manager.go:414] "containerMap: removing container" podUID="432c71ed-404c-4b6d-a521-0ed8ff30f8ea" containerName="payload-pre-processing"
Oct 31 20:02:59.524423 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.524419    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432c71ed-404c-4b6d-a521-0ed8ff30f8ea" containerName="payload-pre-processing"
Oct 31 20:02:59.524709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.524594    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="432c71ed-404c-4b6d-a521-0ed8ff30f8ea" containerName="payload-pre-processing"
Oct 31 20:02:59.530288 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.530258    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:02:59.549003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.548968    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:59.572400 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.572359    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f871dc-fee7-45b8-922a-ae1cbf0491ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww\" (UID: \"27f871dc-fee7-45b8-922a-ae1cbf0491ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:02:59.572560 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.572506    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbghp\" (UniqueName: \"kubernetes.io/projected/27f871dc-fee7-45b8-922a-ae1cbf0491ce-kube-api-access-pbghp\") pod \"payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww\" (UID: \"27f871dc-fee7-45b8-922a-ae1cbf0491ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:02:59.582137 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.582100    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"]
Oct 31 20:02:59.652677 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.634968    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a89a58\""
Oct 31 20:02:59.652677 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.635124    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a89a58-dockercfg-wf5rd\""
Oct 31 20:02:59.678707 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.678631    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbghp\" (UniqueName: \"kubernetes.io/projected/27f871dc-fee7-45b8-922a-ae1cbf0491ce-kube-api-access-pbghp\") pod \"payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww\" (UID: \"27f871dc-fee7-45b8-922a-ae1cbf0491ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:02:59.678914 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.678887    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f871dc-fee7-45b8-922a-ae1cbf0491ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww\" (UID: \"27f871dc-fee7-45b8-922a-ae1cbf0491ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:02:59.685251 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.685200    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f871dc-fee7-45b8-922a-ae1cbf0491ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww\" (UID: \"27f871dc-fee7-45b8-922a-ae1cbf0491ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:02:59.692323 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.692267    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"]
Oct 31 20:02:59.718304 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.718265    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:02:59.742625 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.742585    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbghp\" (UniqueName: \"kubernetes.io/projected/27f871dc-fee7-45b8-922a-ae1cbf0491ce-kube-api-access-pbghp\") pod \"payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww\" (UID: \"27f871dc-fee7-45b8-922a-ae1cbf0491ce\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:02:59.743767 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.743736    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"]
Oct 31 20:02:59.780321 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.780219    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4404670-ccda-42bd-a961-fe617cfd4216-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl\" (UID: \"e4404670-ccda-42bd-a961-fe617cfd4216\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:02:59.780321 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.780281    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4xt\" (UniqueName: \"kubernetes.io/projected/e4404670-ccda-42bd-a961-fe617cfd4216-kube-api-access-sj4xt\") pod \"payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl\" (UID: \"e4404670-ccda-42bd-a961-fe617cfd4216\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:02:59.845982 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.845940    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:02:59.881634 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.881600    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4404670-ccda-42bd-a961-fe617cfd4216-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl\" (UID: \"e4404670-ccda-42bd-a961-fe617cfd4216\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:02:59.882609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.882578    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4404670-ccda-42bd-a961-fe617cfd4216-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl\" (UID: \"e4404670-ccda-42bd-a961-fe617cfd4216\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:02:59.882755 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.882644    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4xt\" (UniqueName: \"kubernetes.io/projected/e4404670-ccda-42bd-a961-fe617cfd4216-kube-api-access-sj4xt\") pod \"payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl\" (UID: \"e4404670-ccda-42bd-a961-fe617cfd4216\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:02:59.940974 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:02:59.940926    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4xt\" (UniqueName: \"kubernetes.io/projected/e4404670-ccda-42bd-a961-fe617cfd4216-kube-api-access-sj4xt\") pod \"payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl\" (UID: \"e4404670-ccda-42bd-a961-fe617cfd4216\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:03:00.067669 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.067551    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:03:00.267597 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.267554    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:00.283736 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.282970    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"]
Oct 31 20:03:00.286925 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:03:00.286887    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f871dc_fee7_45b8_922a_ae1cbf0491ce.slice/crio-38e53bb862411d42c090d6a78d8a8e279646bce53d9c904acd95d5892d1dae1d WatchSource:0}: Error finding container 38e53bb862411d42c090d6a78d8a8e279646bce53d9c904acd95d5892d1dae1d: Status 404 returned error can't find the container with id 38e53bb862411d42c090d6a78d8a8e279646bce53d9c904acd95d5892d1dae1d
Oct 31 20:03:00.544591 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:03:00.544558    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4404670_ccda_42bd_a961_fe617cfd4216.slice/crio-c28ab309d0f14e131c7a85abc317cf8bd12a41e90e6c4cb5768ff7ee1390459d WatchSource:0}: Error finding container c28ab309d0f14e131c7a85abc317cf8bd12a41e90e6c4cb5768ff7ee1390459d: Status 404 returned error can't find the container with id c28ab309d0f14e131c7a85abc317cf8bd12a41e90e6c4cb5768ff7ee1390459d
Oct 31 20:03:00.547882 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.547854    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"]
Oct 31 20:03:00.875783 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.875678    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww" event={"ID":"27f871dc-fee7-45b8-922a-ae1cbf0491ce","Type":"ContainerStarted","Data":"e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2"}
Oct 31 20:03:00.875783 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.875748    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww" event={"ID":"27f871dc-fee7-45b8-922a-ae1cbf0491ce","Type":"ContainerStarted","Data":"38e53bb862411d42c090d6a78d8a8e279646bce53d9c904acd95d5892d1dae1d"}
Oct 31 20:03:00.876008 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.875886    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:03:00.877603 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.877573    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl" event={"ID":"e4404670-ccda-42bd-a961-fe617cfd4216","Type":"ContainerStarted","Data":"463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7"}
Oct 31 20:03:00.877764 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.877615    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl" event={"ID":"e4404670-ccda-42bd-a961-fe617cfd4216","Type":"ContainerStarted","Data":"c28ab309d0f14e131c7a85abc317cf8bd12a41e90e6c4cb5768ff7ee1390459d"}
Oct 31 20:03:00.877764 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.877680    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:03:00.993918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:00.993863    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww" podStartSLOduration=1.993847122 podStartE2EDuration="1.993847122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:00.990479314 +0000 UTC m=+2799.520013366" watchObservedRunningTime="2026-10-31 20:03:00.993847122 +0000 UTC m=+2799.523381175"
Oct 31 20:03:01.111514 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:01.111457    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl" podStartSLOduration=2.111438403 podStartE2EDuration="2.111438403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:01.106920446 +0000 UTC m=+2799.636454501" watchObservedRunningTime="2026-10-31 20:03:01.111438403 +0000 UTC m=+2799.640972456"
Oct 31 20:03:03.444190 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:03.444149    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:03.749667 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:03.749561    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:04.211463 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:04.211430    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:05.396768 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:03:05.396724    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:03:05.784147 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:05.784052    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:03:06.016535 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:06.016470    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:06.272476 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:06.272362    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:06.533907 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:06.533806    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:06.543298 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:06.543255    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"]
Oct 31 20:03:06.547955 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:06.547919    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-650cf3-8bgl5"]
Oct 31 20:03:06.937511 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:06.937452    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:07.600267 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:07.600230    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e76037-9673-4b7b-bd7d-1b12bc1d6abd" path="/var/lib/kubelet/pods/27e76037-9673-4b7b-bd7d-1b12bc1d6abd/volumes"
Oct 31 20:03:10.662146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.662102    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"]
Oct 31 20:03:10.667957 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.667928    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:10.735384 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.735347    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"]
Oct 31 20:03:10.744520 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.740859    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3063f26-wsk6x\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:10.744520 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.740978    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmmd\" (UniqueName: \"kubernetes.io/projected/32057031-e5f3-4184-924d-7ba28c0d3e5f-kube-api-access-pxmmd\") pod \"maas-api-revoke-sub-b3063f26-wsk6x\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:10.744520 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.741076    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3063f26-wsk6x\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:10.786675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.779915    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:03:10.786675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.780097    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:03:10.786675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.780156    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:03:10.786675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.780877    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:03:10.846318 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.846283    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmmd\" (UniqueName: \"kubernetes.io/projected/32057031-e5f3-4184-924d-7ba28c0d3e5f-kube-api-access-pxmmd\") pod \"maas-api-revoke-sub-b3063f26-wsk6x\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:10.846531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.846360    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3063f26-wsk6x\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:10.846531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.846492    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3063f26-wsk6x\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:10.847166 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.847141    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3063f26-wsk6x\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:10.849119 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.849090    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3063f26-wsk6x\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:10.904356 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.904318    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmmd\" (UniqueName: \"kubernetes.io/projected/32057031-e5f3-4184-924d-7ba28c0d3e5f-kube-api-access-pxmmd\") pod \"maas-api-revoke-sub-b3063f26-wsk6x\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:10.983045 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:10.982945    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:11.363349 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:03:11.363311    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32057031_e5f3_4184_924d_7ba28c0d3e5f.slice/crio-324643b7b4ddc63823258ea95e744353e40debeb7e102202c2e8d47ddb6e1950 WatchSource:0}: Error finding container 324643b7b4ddc63823258ea95e744353e40debeb7e102202c2e8d47ddb6e1950: Status 404 returned error can't find the container with id 324643b7b4ddc63823258ea95e744353e40debeb7e102202c2e8d47ddb6e1950
Oct 31 20:03:11.365823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:11.365797    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"]
Oct 31 20:03:11.664081 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:11.663965    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:11.888216 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:11.888186    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:03:11.888392 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:11.888260    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:03:11.891497 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:11.891467    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:11.971158 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:11.971047    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x" event={"ID":"32057031-e5f3-4184-924d-7ba28c0d3e5f","Type":"ContainerStarted","Data":"5a9266902e7dac8db67c5bfb866ea585d44bc0422a2a7c84ae74ab4d102379a1"}
Oct 31 20:03:11.971358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:11.971166    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x" event={"ID":"32057031-e5f3-4184-924d-7ba28c0d3e5f","Type":"ContainerStarted","Data":"324643b7b4ddc63823258ea95e744353e40debeb7e102202c2e8d47ddb6e1950"}
Oct 31 20:03:12.231612 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:12.231482    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x" podStartSLOduration=2.231457962 podStartE2EDuration="2.231457962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:12.226212857 +0000 UTC m=+2810.755746923" watchObservedRunningTime="2026-10-31 20:03:12.231457962 +0000 UTC m=+2810.760992015"
Oct 31 20:03:12.980565 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:12.980528    5006 generic.go:356] "Generic (PLEG): container finished" podID="32057031-e5f3-4184-924d-7ba28c0d3e5f" containerID="5a9266902e7dac8db67c5bfb866ea585d44bc0422a2a7c84ae74ab4d102379a1" exitCode=0
Oct 31 20:03:12.981024 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:12.980614    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x" event={"ID":"32057031-e5f3-4184-924d-7ba28c0d3e5f","Type":"ContainerDied","Data":"5a9266902e7dac8db67c5bfb866ea585d44bc0422a2a7c84ae74ab4d102379a1"}
Oct 31 20:03:13.103511 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:13.103476    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:13.334083 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:13.333949    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:13.551034 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:13.550994    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:13.889901 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:13.889843    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:14.165577 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.165536    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:14.295494 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.295460    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32057031-e5f3-4184-924d-7ba28c0d3e5f-kube-api-access-pxmmd\" (UniqueName: \"kubernetes.io/projected/32057031-e5f3-4184-924d-7ba28c0d3e5f-kube-api-access-pxmmd\") pod \"32057031-e5f3-4184-924d-7ba28c0d3e5f\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") "
Oct 31 20:03:14.295764 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.295526    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-xks-ca\") pod \"32057031-e5f3-4184-924d-7ba28c0d3e5f\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") "
Oct 31 20:03:14.295764 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.295642    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-openshift-ca\") pod \"32057031-e5f3-4184-924d-7ba28c0d3e5f\" (UID: \"32057031-e5f3-4184-924d-7ba28c0d3e5f\") "
Oct 31 20:03:14.296096 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.296069    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-openshift-ca" pod "32057031-e5f3-4184-924d-7ba28c0d3e5f" (UID: "32057031-e5f3-4184-924d-7ba28c0d3e5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:14.296365 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.296342    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:14.298362 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.298331    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32057031-e5f3-4184-924d-7ba28c0d3e5f-kube-api-access-pxmmd" pod "32057031-e5f3-4184-924d-7ba28c0d3e5f" (UID: "32057031-e5f3-4184-924d-7ba28c0d3e5f"). InnerVolumeSpecName "kube-api-access-pxmmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:14.300827 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.300790    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-xks-ca" pod "32057031-e5f3-4184-924d-7ba28c0d3e5f" (UID: "32057031-e5f3-4184-924d-7ba28c0d3e5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:14.341793 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.341759    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:14.398019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.397924    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxmmd\" (UniqueName: \"kubernetes.io/projected/32057031-e5f3-4184-924d-7ba28c0d3e5f-kube-api-access-pxmmd\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:14.398019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.397958    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32057031-e5f3-4184-924d-7ba28c0d3e5f-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:14.569246 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.569209    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:14.998147 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.998108    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"
Oct 31 20:03:14.998313 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.998122    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x" event={"ID":"32057031-e5f3-4184-924d-7ba28c0d3e5f","Type":"ContainerDied","Data":"324643b7b4ddc63823258ea95e744353e40debeb7e102202c2e8d47ddb6e1950"}
Oct 31 20:03:14.998313 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:14.998220    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="324643b7b4ddc63823258ea95e744353e40debeb7e102202c2e8d47ddb6e1950"
Oct 31 20:03:15.227556 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:15.227521    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:15.481645 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:15.481602    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:17.119867 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:17.119828    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:19.516623 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:19.516589    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:19.747883 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:19.747850    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:19.971449 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:19.971415    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:21.433239 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:21.433201    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:22.240455 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:22.240422    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:22.826017 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:22.808718    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:23.059423 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.059383    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:23.269195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.269145    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"]
Oct 31 20:03:23.269816 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.269799    5006 cpu_manager.go:414] "containerMap: removing container" podUID="32057031-e5f3-4184-924d-7ba28c0d3e5f" containerName="revoke-keys"
Oct 31 20:03:23.269878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.269817    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32057031-e5f3-4184-924d-7ba28c0d3e5f" containerName="revoke-keys"
Oct 31 20:03:23.269917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.269908    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="32057031-e5f3-4184-924d-7ba28c0d3e5f" containerName="revoke-keys"
Oct 31 20:03:23.273701 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.273626    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:03:23.343673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.336934    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"]
Oct 31 20:03:23.343673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.341942    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e1347c-23bb-4951-aa24-0117b25e05df-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6\" (UID: \"f5e1347c-23bb-4951-aa24-0117b25e05df\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:03:23.343673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.342050    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k2wk\" (UniqueName: \"kubernetes.io/projected/f5e1347c-23bb-4951-aa24-0117b25e05df-kube-api-access-8k2wk\") pod \"payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6\" (UID: \"f5e1347c-23bb-4951-aa24-0117b25e05df\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:03:23.377595 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.377550    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5d59a2\""
Oct 31 20:03:23.378862 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.378836    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5d59a2-dockercfg-6tsql\""
Oct 31 20:03:23.429018 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.428979    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"]
Oct 31 20:03:23.433298 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.433274    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:03:23.443581 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.443541    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e1347c-23bb-4951-aa24-0117b25e05df-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6\" (UID: \"f5e1347c-23bb-4951-aa24-0117b25e05df\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:03:23.443760 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.443696    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k2wk\" (UniqueName: \"kubernetes.io/projected/f5e1347c-23bb-4951-aa24-0117b25e05df-kube-api-access-8k2wk\") pod \"payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6\" (UID: \"f5e1347c-23bb-4951-aa24-0117b25e05df\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:03:23.444365 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.444344    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e1347c-23bb-4951-aa24-0117b25e05df-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6\" (UID: \"f5e1347c-23bb-4951-aa24-0117b25e05df\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:03:23.487469 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.487427    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"]
Oct 31 20:03:23.501982 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.501943    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k2wk\" (UniqueName: \"kubernetes.io/projected/f5e1347c-23bb-4951-aa24-0117b25e05df-kube-api-access-8k2wk\") pod \"payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6\" (UID: \"f5e1347c-23bb-4951-aa24-0117b25e05df\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:03:23.544808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.544645    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhrhj\" (UniqueName: \"kubernetes.io/projected/044047a1-281a-40e7-954b-86dc240804a3-kube-api-access-vhrhj\") pod \"payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k\" (UID: \"044047a1-281a-40e7-954b-86dc240804a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:03:23.545022 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.544989    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044047a1-281a-40e7-954b-86dc240804a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k\" (UID: \"044047a1-281a-40e7-954b-86dc240804a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:03:23.590581 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.590536    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:03:23.645955 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.645925    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044047a1-281a-40e7-954b-86dc240804a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k\" (UID: \"044047a1-281a-40e7-954b-86dc240804a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:03:23.646148 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.646011    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhrhj\" (UniqueName: \"kubernetes.io/projected/044047a1-281a-40e7-954b-86dc240804a3-kube-api-access-vhrhj\") pod \"payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k\" (UID: \"044047a1-281a-40e7-954b-86dc240804a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:03:23.646912 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.646881    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044047a1-281a-40e7-954b-86dc240804a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k\" (UID: \"044047a1-281a-40e7-954b-86dc240804a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:03:23.705612 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.705336    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhrhj\" (UniqueName: \"kubernetes.io/projected/044047a1-281a-40e7-954b-86dc240804a3-kube-api-access-vhrhj\") pod \"payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k\" (UID: \"044047a1-281a-40e7-954b-86dc240804a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:03:23.749856 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.749819    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:03:23.966791 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:03:23.966760    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e1347c_23bb_4951_aa24_0117b25e05df.slice/crio-a1404b3811f48126412f5d0d80e19104ed773ac105e043b200e0b01c63a41a2b WatchSource:0}: Error finding container a1404b3811f48126412f5d0d80e19104ed773ac105e043b200e0b01c63a41a2b: Status 404 returned error can't find the container with id a1404b3811f48126412f5d0d80e19104ed773ac105e043b200e0b01c63a41a2b
Oct 31 20:03:23.970043 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:23.970022    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"]
Oct 31 20:03:24.070315 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:24.070227    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6" event={"ID":"f5e1347c-23bb-4951-aa24-0117b25e05df","Type":"ContainerStarted","Data":"a1404b3811f48126412f5d0d80e19104ed773ac105e043b200e0b01c63a41a2b"}
Oct 31 20:03:24.142080 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:03:24.142047    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044047a1_281a_40e7_954b_86dc240804a3.slice/crio-72da15fb09f3e28591f67c9273b42abc02369b87149140b0859bbec786af3403 WatchSource:0}: Error finding container 72da15fb09f3e28591f67c9273b42abc02369b87149140b0859bbec786af3403: Status 404 returned error can't find the container with id 72da15fb09f3e28591f67c9273b42abc02369b87149140b0859bbec786af3403
Oct 31 20:03:24.145084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:24.145051    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"]
Oct 31 20:03:25.078628 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:25.078588    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k" event={"ID":"044047a1-281a-40e7-954b-86dc240804a3","Type":"ContainerStarted","Data":"4e0067051329dbf55a820cd4b596427d755faccb5d95f3a28f2fc3d7c0d463a8"}
Oct 31 20:03:25.079265 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:25.078727    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:03:25.079265 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:25.078760    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k" event={"ID":"044047a1-281a-40e7-954b-86dc240804a3","Type":"ContainerStarted","Data":"72da15fb09f3e28591f67c9273b42abc02369b87149140b0859bbec786af3403"}
Oct 31 20:03:25.080535 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:25.080502    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6" event={"ID":"f5e1347c-23bb-4951-aa24-0117b25e05df","Type":"ContainerStarted","Data":"095726f5d506d5c2007d205950900eee1f64ed22626837b58d2e2d5504833fa6"}
Oct 31 20:03:25.080663 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:25.080574    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:03:25.194821 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:25.194769    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k" podStartSLOduration=2.194754912 podStartE2EDuration="2.194754912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:25.19300337 +0000 UTC m=+2823.722537423" watchObservedRunningTime="2026-10-31 20:03:25.194754912 +0000 UTC m=+2823.724288959"
Oct 31 20:03:25.308626 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:25.308591    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:25.310272 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:25.310216    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6" podStartSLOduration=2.310197918 podStartE2EDuration="2.310197918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:25.306970763 +0000 UTC m=+2823.836504842" watchObservedRunningTime="2026-10-31 20:03:25.310197918 +0000 UTC m=+2823.839731971"
Oct 31 20:03:25.602527 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:25.602495    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:26.770980 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.770935    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"]
Oct 31 20:03:26.775304 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.775276    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.831443 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.831404    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7cfafd-openshift-default-dockercfg-96gwp\""
Oct 31 20:03:26.882773 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.882732    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"]
Oct 31 20:03:26.886384 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.886350    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.886514 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.886418    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.886514 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.886492    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.886599 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.886536    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.886681 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.886643    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.886727 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.886694    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.886763 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.886725    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5pr\" (UniqueName: \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-kube-api-access-xd5pr\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.886833 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.886813    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.886893 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.886879    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988242 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988199    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988438 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988341    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988438 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988374    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988438 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988414    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988619 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988461    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988619 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988479    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988619 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988513    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5pr\" (UniqueName: \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-kube-api-access-xd5pr\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988619 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988574    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988886 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988635    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988886 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988667    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988886 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988754    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.988886 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.988812    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.989121 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.989094    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.989419 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.989396    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.991243 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.991215    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:26.991390 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:26.991372    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:27.046279 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:27.046192    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:27.046433 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:27.046310    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5pr\" (UniqueName: \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-kube-api-access-xd5pr\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:27.091557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:27.091509    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:27.310974 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:27.310889    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:27.516150 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:03:27.516112    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2681635f_02e0_4a3f_b674_cf90e4d64f5e.slice/crio-3ac7cbca90f02520ac032b6e9c20d0635f7405648d6cea31cd7bd12ea3988cb1 WatchSource:0}: Error finding container 3ac7cbca90f02520ac032b6e9c20d0635f7405648d6cea31cd7bd12ea3988cb1: Status 404 returned error can't find the container with id 3ac7cbca90f02520ac032b6e9c20d0635f7405648d6cea31cd7bd12ea3988cb1
Oct 31 20:03:27.519217 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:27.519174    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:03:27.519320 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:27.519269    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:03:27.519320 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:27.519314    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:03:27.520491 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:27.520468    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"]
Oct 31 20:03:27.553546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:27.553505    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:27.762944 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:27.762898    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:28.118682 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:28.118260    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" event={"ID":"2681635f-02e0-4a3f-b674-cf90e4d64f5e","Type":"ContainerStarted","Data":"60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb"}
Oct 31 20:03:28.118682 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:28.118333    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" event={"ID":"2681635f-02e0-4a3f-b674-cf90e4d64f5e","Type":"ContainerStarted","Data":"3ac7cbca90f02520ac032b6e9c20d0635f7405648d6cea31cd7bd12ea3988cb1"}
Oct 31 20:03:28.239340 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:28.239278    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" podStartSLOduration=2.239264389 podStartE2EDuration="2.239264389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:28.236747433 +0000 UTC m=+2826.766281505" watchObservedRunningTime="2026-10-31 20:03:28.239264389 +0000 UTC m=+2826.768798442"
Oct 31 20:03:29.092365 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:29.092322    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:29.093892 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:29.093867    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 20:03:29.093975 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:29.093922    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 20:03:30.092106 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:30.092057    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 20:03:30.092541 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:30.092123    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 20:03:31.092786 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:31.092699    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 20:03:31.093166 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:31.092776    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 20:03:32.092666 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:32.092594    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 20:03:32.092885 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:32.092683    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 20:03:32.124779 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:32.122811    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:33.092286 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:33.092243    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 20:03:33.092486 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:33.092303    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 20:03:34.092348 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:34.092305    5006 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 31 20:03:34.092829 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:34.092391    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 31 20:03:35.096835 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:35.096801    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:35.097300 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:35.097284    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:35.098104 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:35.098086    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:03:35.397915 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:03:35.397818    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:03:35.812023 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:35.811931    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:36.091900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:36.091871    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:03:36.092099 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:36.091939    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:03:37.116189 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:37.116154    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:41.972192 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:41.972142    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:46.196532 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:46.196479    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:47.026211 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:47.025936    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:47.356014 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:47.355972    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:48.405938 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:48.405903    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:48.465963 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:48.465919    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:03:48.744905 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:48.744811    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:48.865953 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:48.865908    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"]
Oct 31 20:03:48.908071 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:48.908023    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7cfafd-dhx6v"]
Oct 31 20:03:49.237164 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:49.237132    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:49.457823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:49.457781    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:49.602405 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:49.602369    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78899e07-f80a-4af7-90bf-2be049d010ee" path="/var/lib/kubelet/pods/78899e07-f80a-4af7-90bf-2be049d010ee/volumes"
Oct 31 20:03:49.682980 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:49.682937    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:49.908123 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:49.908025    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:49.974640 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:49.974585    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"]
Oct 31 20:03:50.023571 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:50.023530    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5370f5-mx5w7"]
Oct 31 20:03:50.427207 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:50.427175    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:51.625934 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:51.625893    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e6c134-a769-49d0-a220-40c8a0d27a94" path="/var/lib/kubelet/pods/79e6c134-a769-49d0-a220-40c8a0d27a94/volumes"
Oct 31 20:03:52.937003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:52.936950    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:54.582415 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:54.582376    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:55.208102 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:55.208068    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:55.430626 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:55.430594    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:55.450001 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:55.449971    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"]
Oct 31 20:03:55.450261 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:55.450238    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerName="istio-proxy" containerID="cri-o://60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb" gracePeriod=30
Oct 31 20:03:55.666469 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:55.666433    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:57.690520 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:57.690483    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:57.915239 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:57.915194    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:58.128117 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.128078    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:58.788230 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.788192    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"]
Oct 31 20:03:58.794747 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.794719    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:58.845495 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.845458    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"]
Oct 31 20:03:58.847602 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.847570    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:03:58.885629 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.885586    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:58.885863 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.885681    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:58.885942 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.885912    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5hz7\" (UniqueName: \"kubernetes.io/projected/31a28eb2-0806-4214-b6b2-02c632323d69-kube-api-access-s5hz7\") pod \"maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:58.892379 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.892355    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:03:58.892549 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.892536    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:03:58.892595 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.892537    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:03:58.987044 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.986998    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:58.987237 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.987121    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5hz7\" (UniqueName: \"kubernetes.io/projected/31a28eb2-0806-4214-b6b2-02c632323d69-kube-api-access-s5hz7\") pod \"maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:58.987237 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.987184    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:58.987841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.987813    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:58.990101 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:58.990072    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:59.045176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:59.045097    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5hz7\" (UniqueName: \"kubernetes.io/projected/31a28eb2-0806-4214-b6b2-02c632323d69-kube-api-access-s5hz7\") pod \"maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:59.111379 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:59.111342    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:03:59.483615 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:03:59.483576    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a28eb2_0806_4214_b6b2_02c632323d69.slice/crio-e155e9af91d7fa1239e679bfc42b6b4a22e723b02c49c9886e730291ba4b858d WatchSource:0}: Error finding container e155e9af91d7fa1239e679bfc42b6b4a22e723b02c49c9886e730291ba4b858d: Status 404 returned error can't find the container with id e155e9af91d7fa1239e679bfc42b6b4a22e723b02c49c9886e730291ba4b858d
Oct 31 20:03:59.488375 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:59.488343    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"]
Oct 31 20:03:59.674440 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:03:59.674405    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:00.372388 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.372348    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn" event={"ID":"31a28eb2-0806-4214-b6b2-02c632323d69","Type":"ContainerStarted","Data":"9528e66a4eac064f0171e3b99a9f5f9b2ff3d605feb58ae8acff98798147820e"}
Oct 31 20:04:00.372837 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.372415    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn" event={"ID":"31a28eb2-0806-4214-b6b2-02c632323d69","Type":"ContainerStarted","Data":"e155e9af91d7fa1239e679bfc42b6b4a22e723b02c49c9886e730291ba4b858d"}
Oct 31 20:04:00.488738 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.488684    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn" podStartSLOduration=2.488668521 podStartE2EDuration="2.488668521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:00.484825238 +0000 UTC m=+2859.014359287" watchObservedRunningTime="2026-10-31 20:04:00.488668521 +0000 UTC m=+2859.018202574"
Oct 31 20:04:00.754491 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.754465    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:04:00.814161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814126    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istiod-ca-cert\") pod \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") "
Oct 31 20:04:00.814370 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814226    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-data\") pod \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") "
Oct 31 20:04:00.814370 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814253    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-podinfo\") pod \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") "
Oct 31 20:04:00.814370 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814280    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-certs\") pod \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") "
Oct 31 20:04:00.814370 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814300    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-credential-socket\") pod \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") "
Oct 31 20:04:00.814370 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814339    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-token\" (UniqueName: \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-token\") pod \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") "
Oct 31 20:04:00.814643 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814412    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-envoy\") pod \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") "
Oct 31 20:04:00.814643 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814509    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-data" pod "2681635f-02e0-4a3f-b674-cf90e4d64f5e" (UID: "2681635f-02e0-4a3f-b674-cf90e4d64f5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:00.814643 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814520    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-kube-api-access-xd5pr\" (UniqueName: \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-kube-api-access-xd5pr\") pod \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") "
Oct 31 20:04:00.814643 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814571    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istiod-ca-cert" pod "2681635f-02e0-4a3f-b674-cf90e4d64f5e" (UID: "2681635f-02e0-4a3f-b674-cf90e4d64f5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:00.814643 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.814623    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-socket\") pod \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\" (UID: \"2681635f-02e0-4a3f-b674-cf90e4d64f5e\") "
Oct 31 20:04:00.815286 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.815168    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-certs" pod "2681635f-02e0-4a3f-b674-cf90e4d64f5e" (UID: "2681635f-02e0-4a3f-b674-cf90e4d64f5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:00.815286 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.815208    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-credential-socket" pod "2681635f-02e0-4a3f-b674-cf90e4d64f5e" (UID: "2681635f-02e0-4a3f-b674-cf90e4d64f5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:00.815286 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.815265    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-socket" pod "2681635f-02e0-4a3f-b674-cf90e4d64f5e" (UID: "2681635f-02e0-4a3f-b674-cf90e4d64f5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:00.815471 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.815341    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:00.815471 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.815365    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:00.815471 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.815382    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:00.817278 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.817253    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-token" pod "2681635f-02e0-4a3f-b674-cf90e4d64f5e" (UID: "2681635f-02e0-4a3f-b674-cf90e4d64f5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:00.817416 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.817285    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-kube-api-access-xd5pr" pod "2681635f-02e0-4a3f-b674-cf90e4d64f5e" (UID: "2681635f-02e0-4a3f-b674-cf90e4d64f5e"). InnerVolumeSpecName "kube-api-access-xd5pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:00.817416 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.817359    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-podinfo" pod "2681635f-02e0-4a3f-b674-cf90e4d64f5e" (UID: "2681635f-02e0-4a3f-b674-cf90e4d64f5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:04:00.817562 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.817530    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-envoy" pod "2681635f-02e0-4a3f-b674-cf90e4d64f5e" (UID: "2681635f-02e0-4a3f-b674-cf90e4d64f5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:00.918003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.917903    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:00.918003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.917946    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:00.918003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.917963    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:00.918003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.917977    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:00.918003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.917990    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd5pr\" (UniqueName: \"kubernetes.io/projected/2681635f-02e0-4a3f-b674-cf90e4d64f5e-kube-api-access-xd5pr\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:00.918003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:00.918007    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2681635f-02e0-4a3f-b674-cf90e4d64f5e-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:01.381221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.381185    5006 generic.go:356] "Generic (PLEG): container finished" podID="31a28eb2-0806-4214-b6b2-02c632323d69" containerID="9528e66a4eac064f0171e3b99a9f5f9b2ff3d605feb58ae8acff98798147820e" exitCode=0
Oct 31 20:04:01.381706 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.381269    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn" event={"ID":"31a28eb2-0806-4214-b6b2-02c632323d69","Type":"ContainerDied","Data":"9528e66a4eac064f0171e3b99a9f5f9b2ff3d605feb58ae8acff98798147820e"}
Oct 31 20:04:01.382786 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.382759    5006 generic.go:356] "Generic (PLEG): container finished" podID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerID="60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb" exitCode=0
Oct 31 20:04:01.382946 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.382802    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"
Oct 31 20:04:01.382946 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.382837    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" event={"ID":"2681635f-02e0-4a3f-b674-cf90e4d64f5e","Type":"ContainerDied","Data":"60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb"}
Oct 31 20:04:01.382946 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.382900    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4" event={"ID":"2681635f-02e0-4a3f-b674-cf90e4d64f5e","Type":"ContainerDied","Data":"3ac7cbca90f02520ac032b6e9c20d0635f7405648d6cea31cd7bd12ea3988cb1"}
Oct 31 20:04:01.382946 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.382931    5006 scope.go:122] "RemoveContainer" containerID="60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb"
Oct 31 20:04:01.394522 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.394500    5006 scope.go:122] "RemoveContainer" containerID="60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb"
Oct 31 20:04:01.394888 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:04:01.394866    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb\": container with ID starting with 60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb not found: ID does not exist" containerID="60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb"
Oct 31 20:04:01.394978 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.394897    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb"} err="failed to get container status \"60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb\": rpc error: code = NotFound desc = could not find container \"60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb\": container with ID starting with 60a532ac8bd87cec0a154f135043143ae6a130a7ad154ae345e8f4d1edecf9bb not found: ID does not exist"
Oct 31 20:04:01.610714 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.610674    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"]
Oct 31 20:04:01.613868 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:01.613834    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-qk2n4"]
Oct 31 20:04:02.569435 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:02.569408    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:04:02.643049 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:02.642949    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-openshift-ca\") pod \"31a28eb2-0806-4214-b6b2-02c632323d69\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") "
Oct 31 20:04:02.643049 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:02.643040    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-xks-ca\") pod \"31a28eb2-0806-4214-b6b2-02c632323d69\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") "
Oct 31 20:04:02.643286 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:02.643256    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a28eb2-0806-4214-b6b2-02c632323d69-kube-api-access-s5hz7\" (UniqueName: \"kubernetes.io/projected/31a28eb2-0806-4214-b6b2-02c632323d69-kube-api-access-s5hz7\") pod \"31a28eb2-0806-4214-b6b2-02c632323d69\" (UID: \"31a28eb2-0806-4214-b6b2-02c632323d69\") "
Oct 31 20:04:02.643379 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:02.643350    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-openshift-ca" pod "31a28eb2-0806-4214-b6b2-02c632323d69" (UID: "31a28eb2-0806-4214-b6b2-02c632323d69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:02.643769 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:02.643748    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:02.645209 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:02.645182    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-xks-ca" pod "31a28eb2-0806-4214-b6b2-02c632323d69" (UID: "31a28eb2-0806-4214-b6b2-02c632323d69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:02.645384 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:02.645365    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a28eb2-0806-4214-b6b2-02c632323d69-kube-api-access-s5hz7" pod "31a28eb2-0806-4214-b6b2-02c632323d69" (UID: "31a28eb2-0806-4214-b6b2-02c632323d69"). InnerVolumeSpecName "kube-api-access-s5hz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:02.744449 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:02.744416    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a28eb2-0806-4214-b6b2-02c632323d69-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:02.744449 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:02.744446    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5hz7\" (UniqueName: \"kubernetes.io/projected/31a28eb2-0806-4214-b6b2-02c632323d69-kube-api-access-s5hz7\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:03.399804 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:03.399770    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn" event={"ID":"31a28eb2-0806-4214-b6b2-02c632323d69","Type":"ContainerDied","Data":"e155e9af91d7fa1239e679bfc42b6b4a22e723b02c49c9886e730291ba4b858d"}
Oct 31 20:04:03.399994 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:03.399830    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e155e9af91d7fa1239e679bfc42b6b4a22e723b02c49c9886e730291ba4b858d"
Oct 31 20:04:03.399994 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:03.399781    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"
Oct 31 20:04:03.598884 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:03.598847    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" path="/var/lib/kubelet/pods/2681635f-02e0-4a3f-b674-cf90e4d64f5e/volumes"
Oct 31 20:04:04.737403 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:04.737342    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:04.974110 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:04.974067    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:05.202290 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:05.202247    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:05.394872 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:04:05.394831    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:04:06.146332 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:06.146290    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:06.362938 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:06.362900    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:07.983843 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:07.983804    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"]
Oct 31 20:04:07.984403 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:07.984389    5006 cpu_manager.go:414] "containerMap: removing container" podUID="31a28eb2-0806-4214-b6b2-02c632323d69" containerName="revoke-keys"
Oct 31 20:04:07.984449 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:07.984404    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a28eb2-0806-4214-b6b2-02c632323d69" containerName="revoke-keys"
Oct 31 20:04:07.984449 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:07.984437    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerName="istio-proxy"
Oct 31 20:04:07.984449 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:07.984442    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerName="istio-proxy"
Oct 31 20:04:07.984555 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:07.984546    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a28eb2-0806-4214-b6b2-02c632323d69" containerName="revoke-keys"
Oct 31 20:04:07.984593 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:07.984562    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2681635f-02e0-4a3f-b674-cf90e4d64f5e" containerName="istio-proxy"
Oct 31 20:04:07.988534 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:07.988507    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.116118 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.116078    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.116304 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.116132    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.116304 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.116150    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.116304 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.116237    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6fx6\" (UniqueName: \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-kube-api-access-w6fx6\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.116437 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.116369    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.116437 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.116420    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.116535 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.116506    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.116594 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.116570    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.116782 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.116630    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.188860 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.188822    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"]
Oct 31 20:04:08.218603 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.218561    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.218846 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.218631    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.218846 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.218702    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.218846 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.218732    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6fx6\" (UniqueName: \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-kube-api-access-w6fx6\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.218846 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.218802    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.218846 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.218845    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.219148 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.218915    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.219148 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.218981    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.219148 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.219013    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.219324 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.219152    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.219492 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.219468    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.219565 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.219491    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.219807 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.219783    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.220257 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.220233    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.222764 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.222434    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.222764 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.222696    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.276981 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.276906    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.277124 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.277047    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6fx6\" (UniqueName: \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-kube-api-access-w6fx6\") pod \"maas-default-gateway-openshift-default-58d898bd55-zf78f\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.305077 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.305039    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:08.729878 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:04:08.729840    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f432dd_3da5_4865_99be_9cd2cb7be3a4.slice/crio-888bbaf55c02f5711088578de116f130432e6cab21a63afa25cb11cac7730355 WatchSource:0}: Error finding container 888bbaf55c02f5711088578de116f130432e6cab21a63afa25cb11cac7730355: Status 404 returned error can't find the container with id 888bbaf55c02f5711088578de116f130432e6cab21a63afa25cb11cac7730355
Oct 31 20:04:08.732760 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.732716    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:04:08.732871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.732805    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:04:08.732871 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.732847    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:04:08.733785 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:08.733764    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"]
Oct 31 20:04:09.448497 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:09.448462    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" event={"ID":"47f432dd-3da5-4865-99be-9cd2cb7be3a4","Type":"ContainerStarted","Data":"c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f"}
Oct 31 20:04:09.449057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:09.448528    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" event={"ID":"47f432dd-3da5-4865-99be-9cd2cb7be3a4","Type":"ContainerStarted","Data":"888bbaf55c02f5711088578de116f130432e6cab21a63afa25cb11cac7730355"}
Oct 31 20:04:09.687665 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:09.687599    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" podStartSLOduration=2.687581117 podStartE2EDuration="2.687581117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:09.634890677 +0000 UTC m=+2868.164424732" watchObservedRunningTime="2026-10-31 20:04:09.687581117 +0000 UTC m=+2868.217115170"
Oct 31 20:04:10.318754 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:10.318706    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:10.320844 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:10.320804    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zf78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Oct 31 20:04:10.321003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:10.320876    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Oct 31 20:04:10.391521 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:10.391483    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:11.309676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:11.306112    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zf78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Oct 31 20:04:11.309676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:11.306192    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Oct 31 20:04:12.317039 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:12.308101    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zf78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Oct 31 20:04:12.317039 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:12.308165    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Oct 31 20:04:13.163611 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:13.163568    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:13.308213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:13.308167    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zf78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Oct 31 20:04:13.308537 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:13.308507    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Oct 31 20:04:13.380727 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:13.380638    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:14.310287 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:14.310251    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zf78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Oct 31 20:04:14.310602 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:14.310575    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Oct 31 20:04:15.306068 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:15.306029    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zf78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Oct 31 20:04:15.306493 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:15.306091    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Oct 31 20:04:15.816144 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:15.816102    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:16.306177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:16.306132    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zf78f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Oct 31 20:04:16.306593 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:16.306206    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Oct 31 20:04:17.361141 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:17.361109    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:17.362091 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:17.362048    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:17.362862 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:17.362838    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:04:21.035567 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:21.035533    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:26.561830 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:26.561790    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:27.886972 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:27.886932    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:31.713866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:31.713822    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:31.952403 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:31.952328    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:32.189004 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:32.188969    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:33.630549 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:33.630497    5006 generic.go:356] "Generic (PLEG): container finished" podID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerID="f84eea95a62cbd4f5bfed203f94c14fe9baaef7fe72aa61f743c8a28b2224982" exitCode=0
Oct 31 20:04:33.631153 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:33.630585    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" event={"ID":"24d8c84b-aefa-4e42-b121-33efa295c4b2","Type":"ContainerDied","Data":"f84eea95a62cbd4f5bfed203f94c14fe9baaef7fe72aa61f743c8a28b2224982"}
Oct 31 20:04:33.631232 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:33.631194    5006 scope.go:122] "RemoveContainer" containerID="f84eea95a62cbd4f5bfed203f94c14fe9baaef7fe72aa61f743c8a28b2224982"
Oct 31 20:04:33.632922 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:33.632889    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:04:33.633015 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:33.632965    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:04:33.633015 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:33.633000    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:04:34.639940 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:34.639902    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" event={"ID":"24d8c84b-aefa-4e42-b121-33efa295c4b2","Type":"ContainerStarted","Data":"e7e030e0f3d1871ba5a50de6f4bcfa7fb5fd6f075c061fa85d2deea4b8740330"}
Oct 31 20:04:34.959873 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:34.959774    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:04:34.960097 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:34.960075    5006 patch_prober.go:29] interesting pod/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 20:04:34.960147 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:34.960126    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 20:04:35.395460 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:04:35.395420    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:04:35.888552 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:35.888512    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:35.959949 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:35.959911    5006 patch_prober.go:29] interesting pod/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 20:04:35.960147 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:35.959969    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 20:04:36.121682 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:36.119195    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:36.960220 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:36.960179    5006 patch_prober.go:29] interesting pod/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 20:04:36.960639 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:36.960234    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 20:04:37.148635 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.148595    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:37.486471 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.486434    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"]
Oct 31 20:04:37.491365 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.491343    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.543605 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.543568    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"]
Oct 31 20:04:37.590222 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.590173    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn57q\" (UniqueName: \"kubernetes.io/projected/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-kube-api-access-xn57q\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.590222 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.590209    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:04:37.590460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.590222    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:04:37.590460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.590242    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:04:37.590460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.590231    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.590460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.590309    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.594219 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.594196    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:04:37.691414 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.691363    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn57q\" (UniqueName: \"kubernetes.io/projected/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-kube-api-access-xn57q\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.691414 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.691421    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.691728 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.691438    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.692381 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.692352    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.694025 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.694005    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.749696 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.749605    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn57q\" (UniqueName: \"kubernetes.io/projected/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-kube-api-access-xn57q\") pod \"maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.820955 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.820909    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:37.960220 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.960171    5006 patch_prober.go:29] interesting pod/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 20:04:37.960425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:37.960246    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 20:04:38.217673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:38.217359    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"]
Oct 31 20:04:38.673240 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:38.673201    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg" event={"ID":"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187","Type":"ContainerStarted","Data":"358c2ecd085776ca27e9a418293e6f999db860c663a115cbe7070948396b7d93"}
Oct 31 20:04:38.960102 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:38.960007    5006 patch_prober.go:29] interesting pod/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 20:04:38.960265 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:38.960090    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 20:04:39.682831 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:39.682728    5006 generic.go:356] "Generic (PLEG): container finished" podID="ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187" containerID="453bf9dcb3bff573c841f23c4d4de78226619d02bb762d350168edb3780e8d79" exitCode=0
Oct 31 20:04:39.682831 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:39.682818    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg" event={"ID":"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187","Type":"ContainerDied","Data":"453bf9dcb3bff573c841f23c4d4de78226619d02bb762d350168edb3780e8d79"}
Oct 31 20:04:39.825727 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:39.825662    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:04:39.960817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:39.960722    5006 patch_prober.go:29] interesting pod/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 20:04:39.960817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:39.960781    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 20:04:40.873479 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.873444    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:40.884461 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.884424    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-kube-api-access-xn57q\" (UniqueName: \"kubernetes.io/projected/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-kube-api-access-xn57q\") pod \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") "
Oct 31 20:04:40.884588 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.884569    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-openshift-ca\") pod \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") "
Oct 31 20:04:40.884631 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.884621    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-xks-ca\") pod \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\" (UID: \"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187\") "
Oct 31 20:04:40.884968 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.884945    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-openshift-ca" pod "ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187" (UID: "ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:40.885151 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.885133    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:40.886811 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.886788    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-xks-ca" pod "ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187" (UID: "ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:40.886811 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.886797    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-kube-api-access-xn57q" pod "ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187" (UID: "ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187"). InnerVolumeSpecName "kube-api-access-xn57q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:40.965158 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.965073    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:04:40.966208 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.966183    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:04:40.986318 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.986044    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:40.986318 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:40.986082    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn57q\" (UniqueName: \"kubernetes.io/projected/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187-kube-api-access-xn57q\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:41.709633 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:41.709576    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg" event={"ID":"ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187","Type":"ContainerDied","Data":"358c2ecd085776ca27e9a418293e6f999db860c663a115cbe7070948396b7d93"}
Oct 31 20:04:41.709872 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:41.709598    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"
Oct 31 20:04:41.709872 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:41.709720    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="358c2ecd085776ca27e9a418293e6f999db860c663a115cbe7070948396b7d93"
Oct 31 20:04:42.717077 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:42.717035    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:48.019404 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:48.019367    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:49.023760 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:49.023715    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"]
Oct 31 20:04:49.132790 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:49.132753    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"]
Oct 31 20:04:49.470133 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:49.470092    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:49.692245 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:49.692208    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:52.072248 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.072208    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"]
Oct 31 20:04:52.072880 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.072865    5006 cpu_manager.go:414] "containerMap: removing container" podUID="ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187" containerName="revoke-keys"
Oct 31 20:04:52.072934 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.072881    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187" containerName="revoke-keys"
Oct 31 20:04:52.072998 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.072989    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187" containerName="revoke-keys"
Oct 31 20:04:52.076461 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.076442    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.129057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.129022    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:04:52.129057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.129017    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:04:52.129697 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.129676    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"]
Oct 31 20:04:52.131932 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.131902    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ae198e-72cjh\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.132069 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.131975    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ae198e-72cjh\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.132134 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.132111    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjmkm\" (UniqueName: \"kubernetes.io/projected/b91109a4-d18d-483b-872a-ae49dc4d580a-kube-api-access-kjmkm\") pod \"maas-api-revoke-sub-c0ae198e-72cjh\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.176371 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.176329    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:04:52.176371 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.176329    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:04:52.233565 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.233506    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ae198e-72cjh\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.233829 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.233609    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ae198e-72cjh\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.233829 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.233705    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjmkm\" (UniqueName: \"kubernetes.io/projected/b91109a4-d18d-483b-872a-ae49dc4d580a-kube-api-access-kjmkm\") pod \"maas-api-revoke-sub-c0ae198e-72cjh\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.234425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.234396    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ae198e-72cjh\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.236204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.236181    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ae198e-72cjh\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.291289 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.291260    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjmkm\" (UniqueName: \"kubernetes.io/projected/b91109a4-d18d-483b-872a-ae49dc4d580a-kube-api-access-kjmkm\") pod \"maas-api-revoke-sub-c0ae198e-72cjh\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.391180 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.391139    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:52.762218 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:04:52.762186    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91109a4_d18d_483b_872a_ae49dc4d580a.slice/crio-290c324afbc9c37ea0cec516c692c7f6e9ff8baa9dda86d7888bc82cb413f76d WatchSource:0}: Error finding container 290c324afbc9c37ea0cec516c692c7f6e9ff8baa9dda86d7888bc82cb413f76d: Status 404 returned error can't find the container with id 290c324afbc9c37ea0cec516c692c7f6e9ff8baa9dda86d7888bc82cb413f76d
Oct 31 20:04:52.765230 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.765201    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"]
Oct 31 20:04:52.792494 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:52.792454    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh" event={"ID":"b91109a4-d18d-483b-872a-ae49dc4d580a","Type":"ContainerStarted","Data":"290c324afbc9c37ea0cec516c692c7f6e9ff8baa9dda86d7888bc82cb413f76d"}
Oct 31 20:04:53.464453 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:53.464410    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:53.802892 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:53.802789    5006 generic.go:356] "Generic (PLEG): container finished" podID="b91109a4-d18d-483b-872a-ae49dc4d580a" containerID="72a78bbaa0a35d9367bac0934d3eeb5bf969aedc03182f224f4c47efffd19f93" exitCode=0
Oct 31 20:04:53.803078 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:53.802878    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh" event={"ID":"b91109a4-d18d-483b-872a-ae49dc4d580a","Type":"ContainerDied","Data":"72a78bbaa0a35d9367bac0934d3eeb5bf969aedc03182f224f4c47efffd19f93"}
Oct 31 20:04:54.024943 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.024900    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk" podUID="19e9ccd8-1762-45e1-aa02-b8478d5d59f6" containerName="payload-processing" containerID="cri-o://8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae" gracePeriod=25
Oct 31 20:04:54.133059 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.133014    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52" podUID="3f769cfd-b755-4602-89ca-fd3522116a37" containerName="payload-pre-processing" containerID="cri-o://51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be" gracePeriod=25
Oct 31 20:04:54.318175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.318152    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:04:54.359297 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.359254    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-plugins-config-volume\") pod \"19e9ccd8-1762-45e1-aa02-b8478d5d59f6\" (UID: \"19e9ccd8-1762-45e1-aa02-b8478d5d59f6\") "
Oct 31 20:04:54.359503 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.359439    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-kube-api-access-qrv6d\" (UniqueName: \"kubernetes.io/projected/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-kube-api-access-qrv6d\") pod \"19e9ccd8-1762-45e1-aa02-b8478d5d59f6\" (UID: \"19e9ccd8-1762-45e1-aa02-b8478d5d59f6\") "
Oct 31 20:04:54.359798 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.359765    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-plugins-config-volume" pod "19e9ccd8-1762-45e1-aa02-b8478d5d59f6" (UID: "19e9ccd8-1762-45e1-aa02-b8478d5d59f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:54.360183 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.360161    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:54.362572 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.362538    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-kube-api-access-qrv6d" pod "19e9ccd8-1762-45e1-aa02-b8478d5d59f6" (UID: "19e9ccd8-1762-45e1-aa02-b8478d5d59f6"). InnerVolumeSpecName "kube-api-access-qrv6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:54.454550 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.454516    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:04:54.461892 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.461858    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrv6d\" (UniqueName: \"kubernetes.io/projected/19e9ccd8-1762-45e1-aa02-b8478d5d59f6-kube-api-access-qrv6d\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:54.563263 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.563223    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f769cfd-b755-4602-89ca-fd3522116a37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f769cfd-b755-4602-89ca-fd3522116a37-plugins-config-volume\") pod \"3f769cfd-b755-4602-89ca-fd3522116a37\" (UID: \"3f769cfd-b755-4602-89ca-fd3522116a37\") "
Oct 31 20:04:54.563842 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.563315    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f769cfd-b755-4602-89ca-fd3522116a37-kube-api-access-dwpwp\" (UniqueName: \"kubernetes.io/projected/3f769cfd-b755-4602-89ca-fd3522116a37-kube-api-access-dwpwp\") pod \"3f769cfd-b755-4602-89ca-fd3522116a37\" (UID: \"3f769cfd-b755-4602-89ca-fd3522116a37\") "
Oct 31 20:04:54.563842 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.563780    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f769cfd-b755-4602-89ca-fd3522116a37-plugins-config-volume" pod "3f769cfd-b755-4602-89ca-fd3522116a37" (UID: "3f769cfd-b755-4602-89ca-fd3522116a37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:54.565513 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.565487    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f769cfd-b755-4602-89ca-fd3522116a37-kube-api-access-dwpwp" pod "3f769cfd-b755-4602-89ca-fd3522116a37" (UID: "3f769cfd-b755-4602-89ca-fd3522116a37"). InnerVolumeSpecName "kube-api-access-dwpwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:54.665157 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.665116    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwpwp\" (UniqueName: \"kubernetes.io/projected/3f769cfd-b755-4602-89ca-fd3522116a37-kube-api-access-dwpwp\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:54.665157 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.665152    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f769cfd-b755-4602-89ca-fd3522116a37-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:54.814421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.814324    5006 generic.go:356] "Generic (PLEG): container finished" podID="19e9ccd8-1762-45e1-aa02-b8478d5d59f6" containerID="8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae" exitCode=0
Oct 31 20:04:54.814421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.814378    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"
Oct 31 20:04:54.814632 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.814424    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk" event={"ID":"19e9ccd8-1762-45e1-aa02-b8478d5d59f6","Type":"ContainerDied","Data":"8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae"}
Oct 31 20:04:54.814632 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.814504    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk" event={"ID":"19e9ccd8-1762-45e1-aa02-b8478d5d59f6","Type":"ContainerDied","Data":"0dd631d61ec460779d6aa5a6424f1f0c54e2e940de5c804a1b4926e76051a27b"}
Oct 31 20:04:54.814632 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.814535    5006 scope.go:122] "RemoveContainer" containerID="8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae"
Oct 31 20:04:54.816746 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.816704    5006 generic.go:356] "Generic (PLEG): container finished" podID="3f769cfd-b755-4602-89ca-fd3522116a37" containerID="51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be" exitCode=0
Oct 31 20:04:54.816746 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.816729    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"
Oct 31 20:04:54.816746 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.816739    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52" event={"ID":"3f769cfd-b755-4602-89ca-fd3522116a37","Type":"ContainerDied","Data":"51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be"}
Oct 31 20:04:54.817020 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.816790    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52" event={"ID":"3f769cfd-b755-4602-89ca-fd3522116a37","Type":"ContainerDied","Data":"a741c66380d2f91de4d55c952d75505187d370e8a06cce79835301178d015c75"}
Oct 31 20:04:54.829284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.829259    5006 scope.go:122] "RemoveContainer" containerID="8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae"
Oct 31 20:04:54.829722 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:04:54.829694    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae\": container with ID starting with 8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae not found: ID does not exist" containerID="8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae"
Oct 31 20:04:54.829833 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.829734    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae"} err="failed to get container status \"8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae\": rpc error: code = NotFound desc = could not find container \"8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae\": container with ID starting with 8af668692d0af5a0b66dd37f26cf584b6c67afb9f856c0396426df7ea4b980ae not found: ID does not exist"
Oct 31 20:04:54.829833 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.829762    5006 scope.go:122] "RemoveContainer" containerID="51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be"
Oct 31 20:04:54.841146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.841125    5006 scope.go:122] "RemoveContainer" containerID="51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be"
Oct 31 20:04:54.841472 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:04:54.841447    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be\": container with ID starting with 51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be not found: ID does not exist" containerID="51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be"
Oct 31 20:04:54.841570 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.841485    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be"} err="failed to get container status \"51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be\": rpc error: code = NotFound desc = could not find container \"51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be\": container with ID starting with 51420f2d25c6a662fdd9b888bf42001ddcdb879f9d680a2e79dc9f132b19a9be not found: ID does not exist"
Oct 31 20:04:54.986926 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.986882    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"]
Oct 31 20:04:54.992516 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.992479    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-qwk52"]
Oct 31 20:04:54.996341 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:54.996312    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:55.006226 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.006194    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:04:55.080804 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.080711    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-openshift-ca\") pod \"b91109a4-d18d-483b-872a-ae49dc4d580a\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") "
Oct 31 20:04:55.081032 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.081010    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91109a4-d18d-483b-872a-ae49dc4d580a-kube-api-access-kjmkm\" (UniqueName: \"kubernetes.io/projected/b91109a4-d18d-483b-872a-ae49dc4d580a-kube-api-access-kjmkm\") pod \"b91109a4-d18d-483b-872a-ae49dc4d580a\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") "
Oct 31 20:04:55.081098 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.081083    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-xks-ca\") pod \"b91109a4-d18d-483b-872a-ae49dc4d580a\" (UID: \"b91109a4-d18d-483b-872a-ae49dc4d580a\") "
Oct 31 20:04:55.081237 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.081081    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-openshift-ca" pod "b91109a4-d18d-483b-872a-ae49dc4d580a" (UID: "b91109a4-d18d-483b-872a-ae49dc4d580a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:55.081776 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.081745    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:55.083371 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.083337    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-xks-ca" pod "b91109a4-d18d-483b-872a-ae49dc4d580a" (UID: "b91109a4-d18d-483b-872a-ae49dc4d580a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:55.083515 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.083501    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91109a4-d18d-483b-872a-ae49dc4d580a-kube-api-access-kjmkm" pod "b91109a4-d18d-483b-872a-ae49dc4d580a" (UID: "b91109a4-d18d-483b-872a-ae49dc4d580a"). InnerVolumeSpecName "kube-api-access-kjmkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:55.098860 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.098818    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"]
Oct 31 20:04:55.102596 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.102558    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-bdlhk"]
Oct 31 20:04:55.182869 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.182825    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjmkm\" (UniqueName: \"kubernetes.io/projected/b91109a4-d18d-483b-872a-ae49dc4d580a-kube-api-access-kjmkm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:55.182869 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.182871    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91109a4-d18d-483b-872a-ae49dc4d580a-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:55.599440 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.599404    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e9ccd8-1762-45e1-aa02-b8478d5d59f6" path="/var/lib/kubelet/pods/19e9ccd8-1762-45e1-aa02-b8478d5d59f6/volumes"
Oct 31 20:04:55.599899 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.599860    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f769cfd-b755-4602-89ca-fd3522116a37" path="/var/lib/kubelet/pods/3f769cfd-b755-4602-89ca-fd3522116a37/volumes"
Oct 31 20:04:55.826626 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.826593    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"
Oct 31 20:04:55.826626 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.826618    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh" event={"ID":"b91109a4-d18d-483b-872a-ae49dc4d580a","Type":"ContainerDied","Data":"290c324afbc9c37ea0cec516c692c7f6e9ff8baa9dda86d7888bc82cb413f76d"}
Oct 31 20:04:55.826911 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:55.826705    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="290c324afbc9c37ea0cec516c692c7f6e9ff8baa9dda86d7888bc82cb413f76d"
Oct 31 20:04:58.392395 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:04:58.392358    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:00.362023 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:00.361989    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:00.583546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:00.583506    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:03.023906 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:03.023866    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:03.270979 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:03.270944    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:05.394847 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:05:05.394804    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:05:07.541368 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:07.541326    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:08.921488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:08.921446    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:09.170996 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:09.170955    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:09.392937 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:09.392876    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:09.526095 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:09.526053    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"]
Oct 31 20:05:09.579991 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:09.579943    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13e9b4-xkdqn"]
Oct 31 20:05:09.599732 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:09.599698    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31a28eb2-0806-4214-b6b2-02c632323d69" path="/var/lib/kubelet/pods/31a28eb2-0806-4214-b6b2-02c632323d69/volumes"
Oct 31 20:05:09.610847 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:09.610817    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:09.996095 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:09.996058    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:14.598416 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:14.598380    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:15.886345 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:15.886303    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:16.114289 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:16.114245    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:16.335967 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:16.335936    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:16.744635 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:16.744547    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:17.199580 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.199545    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:17.205881 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.205854    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"]
Oct 31 20:05:17.206455 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.206440    5006 cpu_manager.go:414] "containerMap: removing container" podUID="3f769cfd-b755-4602-89ca-fd3522116a37" containerName="payload-pre-processing"
Oct 31 20:05:17.206498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.206457    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f769cfd-b755-4602-89ca-fd3522116a37" containerName="payload-pre-processing"
Oct 31 20:05:17.206498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.206474    5006 cpu_manager.go:414] "containerMap: removing container" podUID="b91109a4-d18d-483b-872a-ae49dc4d580a" containerName="revoke-keys"
Oct 31 20:05:17.206498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.206480    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91109a4-d18d-483b-872a-ae49dc4d580a" containerName="revoke-keys"
Oct 31 20:05:17.206498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.206494    5006 cpu_manager.go:414] "containerMap: removing container" podUID="19e9ccd8-1762-45e1-aa02-b8478d5d59f6" containerName="payload-processing"
Oct 31 20:05:17.206629 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.206499    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e9ccd8-1762-45e1-aa02-b8478d5d59f6" containerName="payload-processing"
Oct 31 20:05:17.206629 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.206606    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e9ccd8-1762-45e1-aa02-b8478d5d59f6" containerName="payload-processing"
Oct 31 20:05:17.206629 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.206622    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f769cfd-b755-4602-89ca-fd3522116a37" containerName="payload-pre-processing"
Oct 31 20:05:17.206750 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.206630    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91109a4-d18d-483b-872a-ae49dc4d580a" containerName="revoke-keys"
Oct 31 20:05:17.211029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.211010    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.264808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.264771    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"]
Oct 31 20:05:17.310313 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.310282    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:05:17.310313 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.310298    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:05:17.310535 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.310298    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:05:17.314118 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.314095    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:05:17.319897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.319867    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.320020 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.319989    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbnnz\" (UniqueName: \"kubernetes.io/projected/f958db3d-0ef2-46a0-9126-eef121f5e577-kube-api-access-vbnnz\") pod \"maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.320128 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.320111    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.421324 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.421285    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbnnz\" (UniqueName: \"kubernetes.io/projected/f958db3d-0ef2-46a0-9126-eef121f5e577-kube-api-access-vbnnz\") pod \"maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.421538 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.421362    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.421538 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.421485    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.422120 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.422096    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.423997 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.423968    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.479129 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.479057    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbnnz\" (UniqueName: \"kubernetes.io/projected/f958db3d-0ef2-46a0-9126-eef121f5e577-kube-api-access-vbnnz\") pod \"maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.525270 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.525233    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:17.636316 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.636282    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"]
Oct 31 20:05:17.641852 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.641826    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.714837 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.713826    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"]
Oct 31 20:05:17.725471 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.725429    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7gr\" (UniqueName: \"kubernetes.io/projected/188cace0-24fc-4e33-a02f-53e71317cbe0-kube-api-access-ww7gr\") pod \"maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.725714 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.725577    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.725935 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.725907    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.827553 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.827454    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7gr\" (UniqueName: \"kubernetes.io/projected/188cace0-24fc-4e33-a02f-53e71317cbe0-kube-api-access-ww7gr\") pod \"maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.827773 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.827583    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.827773 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.827639    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.828316 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.828296    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.830824 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.830807    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.885168 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.885136    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7gr\" (UniqueName: \"kubernetes.io/projected/188cace0-24fc-4e33-a02f-53e71317cbe0-kube-api-access-ww7gr\") pod \"maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.895283 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:05:17.895251    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf958db3d_0ef2_46a0_9126_eef121f5e577.slice/crio-0e5cdee79ac341bad1e149aec2b11fc772f2538e7676ac603288d83964847ffe WatchSource:0}: Error finding container 0e5cdee79ac341bad1e149aec2b11fc772f2538e7676ac603288d83964847ffe: Status 404 returned error can't find the container with id 0e5cdee79ac341bad1e149aec2b11fc772f2538e7676ac603288d83964847ffe
Oct 31 20:05:17.898960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.898933    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"]
Oct 31 20:05:17.960316 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.960282    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:17.991210 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:17.991178    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv" event={"ID":"f958db3d-0ef2-46a0-9126-eef121f5e577","Type":"ContainerStarted","Data":"0e5cdee79ac341bad1e149aec2b11fc772f2538e7676ac603288d83964847ffe"}
Oct 31 20:05:18.341411 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:05:18.341359    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188cace0_24fc_4e33_a02f_53e71317cbe0.slice/crio-0efc457c76bd72fcd26dcab5e36abee3b6b70b38f861231acb4c5bd1ac62e670 WatchSource:0}: Error finding container 0efc457c76bd72fcd26dcab5e36abee3b6b70b38f861231acb4c5bd1ac62e670: Status 404 returned error can't find the container with id 0efc457c76bd72fcd26dcab5e36abee3b6b70b38f861231acb4c5bd1ac62e670
Oct 31 20:05:18.342510 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:18.342477    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"]
Oct 31 20:05:19.053673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:19.049102    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l" event={"ID":"188cace0-24fc-4e33-a02f-53e71317cbe0","Type":"ContainerStarted","Data":"7a39620afd7c8cf573498c68a3048f1358f3404b5b1fca78b8af7e102c40199b"}
Oct 31 20:05:19.053673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:19.049175    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l" event={"ID":"188cace0-24fc-4e33-a02f-53e71317cbe0","Type":"ContainerStarted","Data":"0efc457c76bd72fcd26dcab5e36abee3b6b70b38f861231acb4c5bd1ac62e670"}
Oct 31 20:05:19.053673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:19.053105    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv" event={"ID":"f958db3d-0ef2-46a0-9126-eef121f5e577","Type":"ContainerStarted","Data":"cc60ef7e357a3e0401aa4b55873b41ed6148c6ae92941842c42ca8cc5dda6561"}
Oct 31 20:05:19.190246 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:19.190179    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l" podStartSLOduration=2.190160017 podStartE2EDuration="2.190160017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:19.189126277 +0000 UTC m=+2937.718660353" watchObservedRunningTime="2026-10-31 20:05:19.190160017 +0000 UTC m=+2937.719694071"
Oct 31 20:05:19.317277 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:19.317126    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv" podStartSLOduration=2.317102461 podStartE2EDuration="2.317102461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:19.311124207 +0000 UTC m=+2937.840658261" watchObservedRunningTime="2026-10-31 20:05:19.317102461 +0000 UTC m=+2937.846636516"
Oct 31 20:05:19.546474 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:05:19.546431    5006 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188cace0_24fc_4e33_a02f_53e71317cbe0.slice/crio-7a39620afd7c8cf573498c68a3048f1358f3404b5b1fca78b8af7e102c40199b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188cace0_24fc_4e33_a02f_53e71317cbe0.slice/crio-conmon-7a39620afd7c8cf573498c68a3048f1358f3404b5b1fca78b8af7e102c40199b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 20:05:20.063425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:20.063379    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:20.066873 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:20.066839    5006 generic.go:356] "Generic (PLEG): container finished" podID="f958db3d-0ef2-46a0-9126-eef121f5e577" containerID="cc60ef7e357a3e0401aa4b55873b41ed6148c6ae92941842c42ca8cc5dda6561" exitCode=0
Oct 31 20:05:20.067046 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:20.066876    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv" event={"ID":"f958db3d-0ef2-46a0-9126-eef121f5e577","Type":"ContainerDied","Data":"cc60ef7e357a3e0401aa4b55873b41ed6148c6ae92941842c42ca8cc5dda6561"}
Oct 31 20:05:20.069737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:20.069705    5006 generic.go:356] "Generic (PLEG): container finished" podID="188cace0-24fc-4e33-a02f-53e71317cbe0" containerID="7a39620afd7c8cf573498c68a3048f1358f3404b5b1fca78b8af7e102c40199b" exitCode=0
Oct 31 20:05:20.069879 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:20.069783    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l" event={"ID":"188cace0-24fc-4e33-a02f-53e71317cbe0","Type":"ContainerDied","Data":"7a39620afd7c8cf573498c68a3048f1358f3404b5b1fca78b8af7e102c40199b"}
Oct 31 20:05:21.253065 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.253035    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:21.278355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.278326    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:21.399383 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.399339    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188cace0-24fc-4e33-a02f-53e71317cbe0-kube-api-access-ww7gr\" (UniqueName: \"kubernetes.io/projected/188cace0-24fc-4e33-a02f-53e71317cbe0-kube-api-access-ww7gr\") pod \"188cace0-24fc-4e33-a02f-53e71317cbe0\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") "
Oct 31 20:05:21.399587 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.399568    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-xks-ca\") pod \"f958db3d-0ef2-46a0-9126-eef121f5e577\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") "
Oct 31 20:05:21.399685 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.399625    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-openshift-ca\") pod \"f958db3d-0ef2-46a0-9126-eef121f5e577\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") "
Oct 31 20:05:21.399757 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.399686    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-openshift-ca\") pod \"188cace0-24fc-4e33-a02f-53e71317cbe0\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") "
Oct 31 20:05:21.399757 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.399731    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f958db3d-0ef2-46a0-9126-eef121f5e577-kube-api-access-vbnnz\" (UniqueName: \"kubernetes.io/projected/f958db3d-0ef2-46a0-9126-eef121f5e577-kube-api-access-vbnnz\") pod \"f958db3d-0ef2-46a0-9126-eef121f5e577\" (UID: \"f958db3d-0ef2-46a0-9126-eef121f5e577\") "
Oct 31 20:05:21.399881 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.399826    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-xks-ca\") pod \"188cace0-24fc-4e33-a02f-53e71317cbe0\" (UID: \"188cace0-24fc-4e33-a02f-53e71317cbe0\") "
Oct 31 20:05:21.400124 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.400091    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-openshift-ca" pod "188cace0-24fc-4e33-a02f-53e71317cbe0" (UID: "188cace0-24fc-4e33-a02f-53e71317cbe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:21.400289 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.400265    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-openshift-ca" pod "f958db3d-0ef2-46a0-9126-eef121f5e577" (UID: "f958db3d-0ef2-46a0-9126-eef121f5e577"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:21.400408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.400381    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.400408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.400407    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.402240 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.402213    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-xks-ca" pod "f958db3d-0ef2-46a0-9126-eef121f5e577" (UID: "f958db3d-0ef2-46a0-9126-eef121f5e577"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:21.402416 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.402393    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-xks-ca" pod "188cace0-24fc-4e33-a02f-53e71317cbe0" (UID: "188cace0-24fc-4e33-a02f-53e71317cbe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:21.402528 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.402510    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188cace0-24fc-4e33-a02f-53e71317cbe0-kube-api-access-ww7gr" pod "188cace0-24fc-4e33-a02f-53e71317cbe0" (UID: "188cace0-24fc-4e33-a02f-53e71317cbe0"). InnerVolumeSpecName "kube-api-access-ww7gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:21.402600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.402537    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f958db3d-0ef2-46a0-9126-eef121f5e577-kube-api-access-vbnnz" pod "f958db3d-0ef2-46a0-9126-eef121f5e577" (UID: "f958db3d-0ef2-46a0-9126-eef121f5e577"). InnerVolumeSpecName "kube-api-access-vbnnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:21.502146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.502110    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f958db3d-0ef2-46a0-9126-eef121f5e577-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.502146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.502141    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbnnz\" (UniqueName: \"kubernetes.io/projected/f958db3d-0ef2-46a0-9126-eef121f5e577-kube-api-access-vbnnz\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.502146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.502152    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188cace0-24fc-4e33-a02f-53e71317cbe0-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.502390 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.502162    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww7gr\" (UniqueName: \"kubernetes.io/projected/188cace0-24fc-4e33-a02f-53e71317cbe0-kube-api-access-ww7gr\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.635102 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.635067    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"]
Oct 31 20:05:21.856937 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:21.856898    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:22.057113 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:22.057071    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"]
Oct 31 20:05:22.068917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:22.068882    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"]
Oct 31 20:05:22.087852 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:22.087822    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"
Oct 31 20:05:22.088035 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:22.087858    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l" event={"ID":"188cace0-24fc-4e33-a02f-53e71317cbe0","Type":"ContainerDied","Data":"0efc457c76bd72fcd26dcab5e36abee3b6b70b38f861231acb4c5bd1ac62e670"}
Oct 31 20:05:22.088035 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:22.087908    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0efc457c76bd72fcd26dcab5e36abee3b6b70b38f861231acb4c5bd1ac62e670"
Oct 31 20:05:22.089509 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:22.089475    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"
Oct 31 20:05:22.089674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:22.089528    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv" event={"ID":"f958db3d-0ef2-46a0-9126-eef121f5e577","Type":"ContainerDied","Data":"0e5cdee79ac341bad1e149aec2b11fc772f2538e7676ac603288d83964847ffe"}
Oct 31 20:05:22.089674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:22.089577    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e5cdee79ac341bad1e149aec2b11fc772f2538e7676ac603288d83964847ffe"
Oct 31 20:05:22.217740 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:22.217384    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:24.091580 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:24.091546    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:24.242771 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:24.242733    5006 scope.go:122] "RemoveContainer" containerID="ab4690125d9423f1c8a8d243a65ad8893f86aca98c3b5ba8503ad3ff41215d12"
Oct 31 20:05:24.254106 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:24.254083    5006 scope.go:122] "RemoveContainer" containerID="0cb8f07df5228efbe224e80c4035f52612a66e4722446fd576baba64a2484b04"
Oct 31 20:05:24.536545 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:24.536460    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:25.239448 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:25.239399    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:25.507763 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:25.507628    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:25.725468 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:25.725427    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:28.215550 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:28.215503    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:28.444877 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:28.444843    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:29.144980 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:29.144947    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:29.362644 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:29.362605    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:29.650253 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:29.650217    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:29.878011 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:29.877977    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:30.146565 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:30.146523    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:31.224872 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.224834    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"]
Oct 31 20:05:31.350932 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.350883    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"]
Oct 31 20:05:31.351570 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.351547    5006 cpu_manager.go:414] "containerMap: removing container" podUID="f958db3d-0ef2-46a0-9126-eef121f5e577" containerName="revoke-keys"
Oct 31 20:05:31.351704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.351571    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f958db3d-0ef2-46a0-9126-eef121f5e577" containerName="revoke-keys"
Oct 31 20:05:31.351704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.351634    5006 cpu_manager.go:414] "containerMap: removing container" podUID="188cace0-24fc-4e33-a02f-53e71317cbe0" containerName="revoke-keys"
Oct 31 20:05:31.351704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.351643    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188cace0-24fc-4e33-a02f-53e71317cbe0" containerName="revoke-keys"
Oct 31 20:05:31.351902 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.351819    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f958db3d-0ef2-46a0-9126-eef121f5e577" containerName="revoke-keys"
Oct 31 20:05:31.351902 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.351847    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="188cace0-24fc-4e33-a02f-53e71317cbe0" containerName="revoke-keys"
Oct 31 20:05:31.356445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.356419    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:31.398148 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.398111    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"]
Oct 31 20:05:31.402533 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.402511    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:31.409005 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.408972    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:05:31.409005 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.408994    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:05:31.409227 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.409068    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:05:31.409227 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.409088    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:05:31.470830 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.470785    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"]
Oct 31 20:05:31.474527 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.474487    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"]
Oct 31 20:05:31.554944 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.554857    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff718e85-l5smr\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:31.554944 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.554907    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnddl\" (UniqueName: \"kubernetes.io/projected/6aa05dcc-89ae-4259-8e79-31b57d63653e-kube-api-access-vnddl\") pod \"maas-api-revoke-sub-17e49e9f-cbv77\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:31.555130 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.555018    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e49e9f-cbv77\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:31.555130 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.555092    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e49e9f-cbv77\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:31.555251 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.555233    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89l8\" (UniqueName: \"kubernetes.io/projected/91a44c69-e1b4-4bab-aafd-ef66056cbed1-kube-api-access-t89l8\") pod \"maas-api-revoke-sub-ff718e85-l5smr\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:31.555347 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.555333    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff718e85-l5smr\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:31.657267 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.657232    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff718e85-l5smr\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:31.657474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.657342    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff718e85-l5smr\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:31.657474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.657435    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnddl\" (UniqueName: \"kubernetes.io/projected/6aa05dcc-89ae-4259-8e79-31b57d63653e-kube-api-access-vnddl\") pod \"maas-api-revoke-sub-17e49e9f-cbv77\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:31.657474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.657472    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e49e9f-cbv77\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:31.657684 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.657522    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e49e9f-cbv77\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:31.657750 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.657696    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89l8\" (UniqueName: \"kubernetes.io/projected/91a44c69-e1b4-4bab-aafd-ef66056cbed1-kube-api-access-t89l8\") pod \"maas-api-revoke-sub-ff718e85-l5smr\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:31.659024 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.658997    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff718e85-l5smr\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:31.660477 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.660447    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e49e9f-cbv77\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:31.678545 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.678506    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff718e85-l5smr\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:31.679336 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.679311    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e49e9f-cbv77\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:31.686190 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.686153    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:31.718477 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.718435    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnddl\" (UniqueName: \"kubernetes.io/projected/6aa05dcc-89ae-4259-8e79-31b57d63653e-kube-api-access-vnddl\") pod \"maas-api-revoke-sub-17e49e9f-cbv77\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:31.726065 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.726035    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89l8\" (UniqueName: \"kubernetes.io/projected/91a44c69-e1b4-4bab-aafd-ef66056cbed1-kube-api-access-t89l8\") pod \"maas-api-revoke-sub-ff718e85-l5smr\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:31.918838 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.918800    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:31.972551 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:31.972514    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:32.017044 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:32.017011    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:32.343011 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:32.342969    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:32.376184 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:32.376148    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"]
Oct 31 20:05:32.444932 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:05:32.444897    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa05dcc_89ae_4259_8e79_31b57d63653e.slice/crio-ae7eb2c2615db6b33629bc5e08e4aa213dc2cf4c5030a982ea4a0ffcb2c740f2 WatchSource:0}: Error finding container ae7eb2c2615db6b33629bc5e08e4aa213dc2cf4c5030a982ea4a0ffcb2c740f2: Status 404 returned error can't find the container with id ae7eb2c2615db6b33629bc5e08e4aa213dc2cf4c5030a982ea4a0ffcb2c740f2
Oct 31 20:05:32.448035 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:32.447911    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"]
Oct 31 20:05:32.569473 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:32.569427    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:32.848694 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:32.848641    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:33.030142 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:33.030045    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"]
Oct 31 20:05:33.045019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:33.041504    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9902ffd9-h5qtg"]
Oct 31 20:05:33.187869 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:33.187832    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr" event={"ID":"91a44c69-e1b4-4bab-aafd-ef66056cbed1","Type":"ContainerStarted","Data":"05fdce719222c840b3fbfa5deb37c88714c98b37bc02ea053b4202920659ddd0"}
Oct 31 20:05:33.188075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:33.187894    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr" event={"ID":"91a44c69-e1b4-4bab-aafd-ef66056cbed1","Type":"ContainerStarted","Data":"a081e80c061dd8e037fa692692527e0fda91c9260d25800c4b8b7c4032bdb3f3"}
Oct 31 20:05:33.189701 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:33.189643    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77" event={"ID":"6aa05dcc-89ae-4259-8e79-31b57d63653e","Type":"ContainerStarted","Data":"352a65a634a4d27a28fa06b7bbaf40f4bfe1353943579f24babfa618e3a43565"}
Oct 31 20:05:33.189838 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:33.189723    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77" event={"ID":"6aa05dcc-89ae-4259-8e79-31b57d63653e","Type":"ContainerStarted","Data":"ae7eb2c2615db6b33629bc5e08e4aa213dc2cf4c5030a982ea4a0ffcb2c740f2"}
Oct 31 20:05:33.304339 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:33.304231    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr" podStartSLOduration=2.304215937 podStartE2EDuration="2.304215937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:33.302799694 +0000 UTC m=+2951.832333771" watchObservedRunningTime="2026-10-31 20:05:33.304215937 +0000 UTC m=+2951.833749990"
Oct 31 20:05:33.422756 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:33.422689    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77" podStartSLOduration=2.422633646 podStartE2EDuration="2.422633646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:33.422571976 +0000 UTC m=+2951.952106048" watchObservedRunningTime="2026-10-31 20:05:33.422633646 +0000 UTC m=+2951.952167698"
Oct 31 20:05:33.471985 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:33.471942    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:33.600104 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:33.600070    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187" path="/var/lib/kubelet/pods/ed1bb6d0-c227-4cda-9a6a-8dc01ecbf187/volumes"
Oct 31 20:05:34.198254 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:34.198207    5006 generic.go:356] "Generic (PLEG): container finished" podID="91a44c69-e1b4-4bab-aafd-ef66056cbed1" containerID="05fdce719222c840b3fbfa5deb37c88714c98b37bc02ea053b4202920659ddd0" exitCode=0
Oct 31 20:05:34.198254 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:34.198244    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr" event={"ID":"91a44c69-e1b4-4bab-aafd-ef66056cbed1","Type":"ContainerDied","Data":"05fdce719222c840b3fbfa5deb37c88714c98b37bc02ea053b4202920659ddd0"}
Oct 31 20:05:34.200027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:34.200002    5006 generic.go:356] "Generic (PLEG): container finished" podID="6aa05dcc-89ae-4259-8e79-31b57d63653e" containerID="352a65a634a4d27a28fa06b7bbaf40f4bfe1353943579f24babfa618e3a43565" exitCode=0
Oct 31 20:05:34.200185 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:34.200088    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77" event={"ID":"6aa05dcc-89ae-4259-8e79-31b57d63653e","Type":"ContainerDied","Data":"352a65a634a4d27a28fa06b7bbaf40f4bfe1353943579f24babfa618e3a43565"}
Oct 31 20:05:35.307026 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.306977    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:35.461750 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:05:35.460806    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:05:35.499895 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.499828    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:35.502481 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.502458    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:35.596355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.596321    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:35.682673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.673243    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-openshift-ca\") pod \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") "
Oct 31 20:05:35.682673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.674835    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa05dcc-89ae-4259-8e79-31b57d63653e-kube-api-access-vnddl\" (UniqueName: \"kubernetes.io/projected/6aa05dcc-89ae-4259-8e79-31b57d63653e-kube-api-access-vnddl\") pod \"6aa05dcc-89ae-4259-8e79-31b57d63653e\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") "
Oct 31 20:05:35.682673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.674890    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-xks-ca\") pod \"6aa05dcc-89ae-4259-8e79-31b57d63653e\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") "
Oct 31 20:05:35.682673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.674929    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-openshift-ca\") pod \"6aa05dcc-89ae-4259-8e79-31b57d63653e\" (UID: \"6aa05dcc-89ae-4259-8e79-31b57d63653e\") "
Oct 31 20:05:35.682673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.675039    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-xks-ca\") pod \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") "
Oct 31 20:05:35.682673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.675098    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a44c69-e1b4-4bab-aafd-ef66056cbed1-kube-api-access-t89l8\" (UniqueName: \"kubernetes.io/projected/91a44c69-e1b4-4bab-aafd-ef66056cbed1-kube-api-access-t89l8\") pod \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\" (UID: \"91a44c69-e1b4-4bab-aafd-ef66056cbed1\") "
Oct 31 20:05:35.682673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.678461    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-openshift-ca" pod "91a44c69-e1b4-4bab-aafd-ef66056cbed1" (UID: "91a44c69-e1b4-4bab-aafd-ef66056cbed1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:35.698710 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.684176    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-openshift-ca" pod "6aa05dcc-89ae-4259-8e79-31b57d63653e" (UID: "6aa05dcc-89ae-4259-8e79-31b57d63653e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:35.741680 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.741628    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-xks-ca" pod "91a44c69-e1b4-4bab-aafd-ef66056cbed1" (UID: "91a44c69-e1b4-4bab-aafd-ef66056cbed1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:35.743311 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.743265    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a44c69-e1b4-4bab-aafd-ef66056cbed1-kube-api-access-t89l8" pod "91a44c69-e1b4-4bab-aafd-ef66056cbed1" (UID: "91a44c69-e1b4-4bab-aafd-ef66056cbed1"). InnerVolumeSpecName "kube-api-access-t89l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:35.743621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.743596    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa05dcc-89ae-4259-8e79-31b57d63653e-kube-api-access-vnddl" pod "6aa05dcc-89ae-4259-8e79-31b57d63653e" (UID: "6aa05dcc-89ae-4259-8e79-31b57d63653e"). InnerVolumeSpecName "kube-api-access-vnddl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:35.754509 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.751836    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-xks-ca" pod "6aa05dcc-89ae-4259-8e79-31b57d63653e" (UID: "6aa05dcc-89ae-4259-8e79-31b57d63653e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:35.777436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.777401    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:35.777436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.777431    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t89l8\" (UniqueName: \"kubernetes.io/projected/91a44c69-e1b4-4bab-aafd-ef66056cbed1-kube-api-access-t89l8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:35.777436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.777442    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a44c69-e1b4-4bab-aafd-ef66056cbed1-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:35.777730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.777452    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnddl\" (UniqueName: \"kubernetes.io/projected/6aa05dcc-89ae-4259-8e79-31b57d63653e-kube-api-access-vnddl\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:35.777730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.777460    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:35.777730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:35.777469    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa05dcc-89ae-4259-8e79-31b57d63653e-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:36.226133 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:36.226081    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9" podUID="c96b94c3-1d84-46a1-8898-d9a32e19816f" containerName="payload-processing" containerID="cri-o://a991994978f70d09c077c14a4ce7513f8b6dcd506d32a9c13ffd52ad2d256c8e" gracePeriod=25
Oct 31 20:05:36.248146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:36.248111    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"
Oct 31 20:05:36.248146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:36.248140    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77" event={"ID":"6aa05dcc-89ae-4259-8e79-31b57d63653e","Type":"ContainerDied","Data":"ae7eb2c2615db6b33629bc5e08e4aa213dc2cf4c5030a982ea4a0ffcb2c740f2"}
Oct 31 20:05:36.248401 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:36.248205    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7eb2c2615db6b33629bc5e08e4aa213dc2cf4c5030a982ea4a0ffcb2c740f2"
Oct 31 20:05:36.249696 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:36.249676    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"
Oct 31 20:05:36.249823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:36.249707    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr" event={"ID":"91a44c69-e1b4-4bab-aafd-ef66056cbed1","Type":"ContainerDied","Data":"a081e80c061dd8e037fa692692527e0fda91c9260d25800c4b8b7c4032bdb3f3"}
Oct 31 20:05:36.249823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:36.249747    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a081e80c061dd8e037fa692692527e0fda91c9260d25800c4b8b7c4032bdb3f3"
Oct 31 20:05:36.656306 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:36.656266    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main" containerID="cri-o://b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c" gracePeriod=45
Oct 31 20:05:36.966674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:36.964589    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9" podUID="c96b94c3-1d84-46a1-8898-d9a32e19816f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:36.978679 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:36.977334    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 20:05:37.079945 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.079894    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main" containerID="cri-o://555092c5217a49251d0217cd1c7948409c78242ab128dc52e0d2bb001207f0cf" gracePeriod=45
Oct 31 20:05:37.096029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.095273    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-model-cache\") pod \"6494229e-f3e7-4798-b867-33dda48a521d\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") "
Oct 31 20:05:37.096029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.095387    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-tmp-dir\") pod \"6494229e-f3e7-4798-b867-33dda48a521d\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") "
Oct 31 20:05:37.096029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.095433    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6494229e-f3e7-4798-b867-33dda48a521d-kube-api-access-r4phg\" (UniqueName: \"kubernetes.io/projected/6494229e-f3e7-4798-b867-33dda48a521d-kube-api-access-r4phg\") pod \"6494229e-f3e7-4798-b867-33dda48a521d\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") "
Oct 31 20:05:37.096029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.095495    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-home\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-home\") pod \"6494229e-f3e7-4798-b867-33dda48a521d\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") "
Oct 31 20:05:37.096029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.095572    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6494229e-f3e7-4798-b867-33dda48a521d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6494229e-f3e7-4798-b867-33dda48a521d-tls-certs\") pod \"6494229e-f3e7-4798-b867-33dda48a521d\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") "
Oct 31 20:05:37.096029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.095856    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-dshm\") pod \"6494229e-f3e7-4798-b867-33dda48a521d\" (UID: \"6494229e-f3e7-4798-b867-33dda48a521d\") "
Oct 31 20:05:37.100883 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.100850    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-model-cache" pod "6494229e-f3e7-4798-b867-33dda48a521d" (UID: "6494229e-f3e7-4798-b867-33dda48a521d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.101072 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.101050    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-tmp-dir" pod "6494229e-f3e7-4798-b867-33dda48a521d" (UID: "6494229e-f3e7-4798-b867-33dda48a521d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.102933 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.102897    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main" containerID="cri-o://f687613fdac29da3434086470ac1e80ca0e798df08f6b2b7e43bc7124d01843a" gracePeriod=45
Oct 31 20:05:37.103555 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.103528    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-home" pod "6494229e-f3e7-4798-b867-33dda48a521d" (UID: "6494229e-f3e7-4798-b867-33dda48a521d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.105416 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.105388    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-dshm" pod "6494229e-f3e7-4798-b867-33dda48a521d" (UID: "6494229e-f3e7-4798-b867-33dda48a521d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.105523 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.105448    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6494229e-f3e7-4798-b867-33dda48a521d-kube-api-access-r4phg" pod "6494229e-f3e7-4798-b867-33dda48a521d" (UID: "6494229e-f3e7-4798-b867-33dda48a521d"). InnerVolumeSpecName "kube-api-access-r4phg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:37.111173 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.111138    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6494229e-f3e7-4798-b867-33dda48a521d-tls-certs" pod "6494229e-f3e7-4798-b867-33dda48a521d" (UID: "6494229e-f3e7-4798-b867-33dda48a521d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:37.201661 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.201246    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.201661 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.201286    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.201661 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.201303    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4phg\" (UniqueName: \"kubernetes.io/projected/6494229e-f3e7-4798-b867-33dda48a521d-kube-api-access-r4phg\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.201661 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.201313    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.201661 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.201322    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6494229e-f3e7-4798-b867-33dda48a521d-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.201661 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.201330    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6494229e-f3e7-4798-b867-33dda48a521d-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.267362 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.267310    5006 generic.go:356] "Generic (PLEG): container finished" podID="6494229e-f3e7-4798-b867-33dda48a521d" containerID="b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c" exitCode=0
Oct 31 20:05:37.267606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.267571    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" event={"ID":"6494229e-f3e7-4798-b867-33dda48a521d","Type":"ContainerDied","Data":"b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c"}
Oct 31 20:05:37.268606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.267626    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb" event={"ID":"6494229e-f3e7-4798-b867-33dda48a521d","Type":"ContainerDied","Data":"05089b386f47e9e9fdb12790966f78f89878f79e7f71e1e79a715ec83809d509"}
Oct 31 20:05:37.268606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.267671    5006 scope.go:122] "RemoveContainer" containerID="b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c"
Oct 31 20:05:37.270022 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.268988    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"
Oct 31 20:05:37.272910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.272729    5006 generic.go:356] "Generic (PLEG): container finished" podID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerID="555092c5217a49251d0217cd1c7948409c78242ab128dc52e0d2bb001207f0cf" exitCode=0
Oct 31 20:05:37.272910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.272839    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" event={"ID":"b5c7c3d9-eda3-432e-91b6-40496d56f48e","Type":"ContainerDied","Data":"555092c5217a49251d0217cd1c7948409c78242ab128dc52e0d2bb001207f0cf"}
Oct 31 20:05:37.275258 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.275099    5006 generic.go:356] "Generic (PLEG): container finished" podID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerID="f687613fdac29da3434086470ac1e80ca0e798df08f6b2b7e43bc7124d01843a" exitCode=0
Oct 31 20:05:37.275258 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.275204    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" event={"ID":"847f87ee-2302-423f-a9f8-5a307a2e7b1a","Type":"ContainerDied","Data":"f687613fdac29da3434086470ac1e80ca0e798df08f6b2b7e43bc7124d01843a"}
Oct 31 20:05:37.293820 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.293619    5006 scope.go:122] "RemoveContainer" containerID="b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c"
Oct 31 20:05:37.299102 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:05:37.299061    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c\": container with ID starting with b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c not found: ID does not exist" containerID="b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c"
Oct 31 20:05:37.299218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.299179    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c"} err="failed to get container status \"b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c\": rpc error: code = NotFound desc = could not find container \"b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c\": container with ID starting with b3acca4b3b022eca8d386f57d2f4e6d9375beddbb5a6babc973c001b66c04b9c not found: ID does not exist"
Oct 31 20:05:37.440851 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.440812    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"]
Oct 31 20:05:37.449107 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.449074    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-570915/simulator-w5-570915-kserve-d4854df47-vt2rb"]
Oct 31 20:05:37.464881 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.464478    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 20:05:37.469713 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.469642    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 20:05:37.603798 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.603764    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6494229e-f3e7-4798-b867-33dda48a521d" path="/var/lib/kubelet/pods/6494229e-f3e7-4798-b867-33dda48a521d/volumes"
Oct 31 20:05:37.618533 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.618496    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-model-cache\") pod \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") "
Oct 31 20:05:37.618823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.618601    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tls-certs\") pod \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") "
Oct 31 20:05:37.618823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.618633    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c7c3d9-eda3-432e-91b6-40496d56f48e-kube-api-access-plcp8\" (UniqueName: \"kubernetes.io/projected/b5c7c3d9-eda3-432e-91b6-40496d56f48e-kube-api-access-plcp8\") pod \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") "
Oct 31 20:05:37.618823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.618718    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-dshm\") pod \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") "
Oct 31 20:05:37.618823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.618743    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-home\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-home\") pod \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") "
Oct 31 20:05:37.618823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.618802    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tmp-dir\") pod \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") "
Oct 31 20:05:37.619122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.618869    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-home\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-home\") pod \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") "
Oct 31 20:05:37.619122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.618906    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-model-cache\") pod \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\" (UID: \"b5c7c3d9-eda3-432e-91b6-40496d56f48e\") "
Oct 31 20:05:37.619122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.618958    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-dshm\") pod \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") "
Oct 31 20:05:37.619122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.619057    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847f87ee-2302-423f-a9f8-5a307a2e7b1a-kube-api-access-zvqd8\" (UniqueName: \"kubernetes.io/projected/847f87ee-2302-423f-a9f8-5a307a2e7b1a-kube-api-access-zvqd8\") pod \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") "
Oct 31 20:05:37.619344 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.619124    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tmp-dir\") pod \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") "
Oct 31 20:05:37.619344 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.619163    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tls-certs\" (UniqueName: \"kubernetes.io/secret/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tls-certs\") pod \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\" (UID: \"847f87ee-2302-423f-a9f8-5a307a2e7b1a\") "
Oct 31 20:05:37.619622 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.619461    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tmp-dir" pod "b5c7c3d9-eda3-432e-91b6-40496d56f48e" (UID: "b5c7c3d9-eda3-432e-91b6-40496d56f48e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.619767 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.619742    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-home" pod "847f87ee-2302-423f-a9f8-5a307a2e7b1a" (UID: "847f87ee-2302-423f-a9f8-5a307a2e7b1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.620280 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.620077    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-home" pod "b5c7c3d9-eda3-432e-91b6-40496d56f48e" (UID: "b5c7c3d9-eda3-432e-91b6-40496d56f48e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.620280 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.620154    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-model-cache" pod "b5c7c3d9-eda3-432e-91b6-40496d56f48e" (UID: "b5c7c3d9-eda3-432e-91b6-40496d56f48e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.621002 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.620974    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tmp-dir" pod "847f87ee-2302-423f-a9f8-5a307a2e7b1a" (UID: "847f87ee-2302-423f-a9f8-5a307a2e7b1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.621099 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.621039    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-model-cache" pod "847f87ee-2302-423f-a9f8-5a307a2e7b1a" (UID: "847f87ee-2302-423f-a9f8-5a307a2e7b1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.623327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.623282    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-dshm" pod "847f87ee-2302-423f-a9f8-5a307a2e7b1a" (UID: "847f87ee-2302-423f-a9f8-5a307a2e7b1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.623415 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.623327    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847f87ee-2302-423f-a9f8-5a307a2e7b1a-kube-api-access-zvqd8" pod "847f87ee-2302-423f-a9f8-5a307a2e7b1a" (UID: "847f87ee-2302-423f-a9f8-5a307a2e7b1a"). InnerVolumeSpecName "kube-api-access-zvqd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:37.623415 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.623387    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tls-certs" pod "b5c7c3d9-eda3-432e-91b6-40496d56f48e" (UID: "b5c7c3d9-eda3-432e-91b6-40496d56f48e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:37.624316 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.624292    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.624425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.624321    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.624425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.624336    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.624425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.624348    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.624425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.624362    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.624425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.624375    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.624425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.624388    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.624425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.624400    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847f87ee-2302-423f-a9f8-5a307a2e7b1a-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.624425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.624414    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqd8\" (UniqueName: \"kubernetes.io/projected/847f87ee-2302-423f-a9f8-5a307a2e7b1a-kube-api-access-zvqd8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.625718 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.625687    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tls-certs" pod "847f87ee-2302-423f-a9f8-5a307a2e7b1a" (UID: "847f87ee-2302-423f-a9f8-5a307a2e7b1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:37.626152 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.626133    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c7c3d9-eda3-432e-91b6-40496d56f48e-kube-api-access-plcp8" pod "b5c7c3d9-eda3-432e-91b6-40496d56f48e" (UID: "b5c7c3d9-eda3-432e-91b6-40496d56f48e"). InnerVolumeSpecName "kube-api-access-plcp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:37.626488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.626466    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-dshm" pod "b5c7c3d9-eda3-432e-91b6-40496d56f48e" (UID: "b5c7c3d9-eda3-432e-91b6-40496d56f48e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:37.725323 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.725233    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f87ee-2302-423f-a9f8-5a307a2e7b1a-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.725323 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.725263    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plcp8\" (UniqueName: \"kubernetes.io/projected/b5c7c3d9-eda3-432e-91b6-40496d56f48e-kube-api-access-plcp8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.725323 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.725274    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c7c3d9-eda3-432e-91b6-40496d56f48e-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:37.939943 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.939892    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:37.985116 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985023    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"]
Oct 31 20:05:37.985681 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985636    5006 cpu_manager.go:414] "containerMap: removing container" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main"
Oct 31 20:05:37.985681 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985663    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main"
Oct 31 20:05:37.985681 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985676    5006 cpu_manager.go:414] "containerMap: removing container" podUID="91a44c69-e1b4-4bab-aafd-ef66056cbed1" containerName="revoke-keys"
Oct 31 20:05:37.985798 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985683    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a44c69-e1b4-4bab-aafd-ef66056cbed1" containerName="revoke-keys"
Oct 31 20:05:37.985798 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985694    5006 cpu_manager.go:414] "containerMap: removing container" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main"
Oct 31 20:05:37.985798 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985700    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main"
Oct 31 20:05:37.985798 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985721    5006 cpu_manager.go:414] "containerMap: removing container" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main"
Oct 31 20:05:37.985798 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985727    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main"
Oct 31 20:05:37.985798 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985757    5006 cpu_manager.go:414] "containerMap: removing container" podUID="6aa05dcc-89ae-4259-8e79-31b57d63653e" containerName="revoke-keys"
Oct 31 20:05:37.985798 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985762    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa05dcc-89ae-4259-8e79-31b57d63653e" containerName="revoke-keys"
Oct 31 20:05:37.986000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985855    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa05dcc-89ae-4259-8e79-31b57d63653e" containerName="revoke-keys"
Oct 31 20:05:37.986000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985863    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" containerName="main"
Oct 31 20:05:37.986000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985870    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a44c69-e1b4-4bab-aafd-ef66056cbed1" containerName="revoke-keys"
Oct 31 20:05:37.986000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985878    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6494229e-f3e7-4798-b867-33dda48a521d" containerName="main"
Oct 31 20:05:37.986000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.985885    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" containerName="main"
Oct 31 20:05:37.990839 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:37.990818    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.045492 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.045453    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"]
Oct 31 20:05:38.131114 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.131080    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71847a6-27eb-4639-abba-6c9e62a0e15a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.131293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.131128    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.131293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.131255    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.131293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.131289    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.131411 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.131308    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.131411 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.131369    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.131510 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.131488    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.131550 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.131530    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk85l\" (UniqueName: \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-kube-api-access-pk85l\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.131699 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.131686    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.232730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.232700    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.232914 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.232740    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk85l\" (UniqueName: \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-kube-api-access-pk85l\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.232914 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.232817    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.232914 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.232868    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71847a6-27eb-4639-abba-6c9e62a0e15a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.232914 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.232897    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.233156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.232983    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.233156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.233025    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.233156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.233049    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.233156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.233083    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.233156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.233099    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.233434 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.233256    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.233434 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.233384    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.233510 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.233481    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.233673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.233640    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71847a6-27eb-4639-abba-6c9e62a0e15a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.235330 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.235277    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.235521 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.235505    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.285284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.285254    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"
Oct 31 20:05:38.285284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.285276    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t" event={"ID":"b5c7c3d9-eda3-432e-91b6-40496d56f48e","Type":"ContainerDied","Data":"07f318708682646bca32ba864af218afb931ce55ddcd4234233445bb7a3fa88f"}
Oct 31 20:05:38.285560 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.285328    5006 scope.go:122] "RemoveContainer" containerID="555092c5217a49251d0217cd1c7948409c78242ab128dc52e0d2bb001207f0cf"
Oct 31 20:05:38.287109 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.287078    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"
Oct 31 20:05:38.287109 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.287097    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv" event={"ID":"847f87ee-2302-423f-a9f8-5a307a2e7b1a","Type":"ContainerDied","Data":"fcc4d95cd1139b72d32f011cb628e350cf8d2b55b256e6df827fef869d0a6ecb"}
Oct 31 20:05:38.295222 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.295187    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.295372 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.295304    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk85l\" (UniqueName: \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-kube-api-access-pk85l\") pod \"maas-default-gateway-openshift-default-58d898bd55-gjb7m\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.297187 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.297168    5006 scope.go:122] "RemoveContainer" containerID="f687613fdac29da3434086470ac1e80ca0e798df08f6b2b7e43bc7124d01843a"
Oct 31 20:05:38.305887 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.305862    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:38.455115 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.455057    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"]
Oct 31 20:05:38.460778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.460744    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/premium-w1-b5155c-kserve-787d44c58d-4qg2t"]
Oct 31 20:05:38.567326 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.567238    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"]
Oct 31 20:05:38.570767 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.570740    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/simulator-w1-b5155c-kserve-5cbc7b8997-94jsv"]
Oct 31 20:05:38.727215 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:05:38.727140    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71847a6_27eb_4639_abba_6c9e62a0e15a.slice/crio-9d7ba09d5e01eecb2c09a0a7136026665ee57931b589553baade1d86bbf8ce15 WatchSource:0}: Error finding container 9d7ba09d5e01eecb2c09a0a7136026665ee57931b589553baade1d86bbf8ce15: Status 404 returned error can't find the container with id 9d7ba09d5e01eecb2c09a0a7136026665ee57931b589553baade1d86bbf8ce15
Oct 31 20:05:38.729885 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.729851    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:38.730009 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.729920    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:38.730009 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.729947    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:38.730188 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:38.730167    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"]
Oct 31 20:05:39.299876 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:39.299841    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" event={"ID":"d71847a6-27eb-4639-abba-6c9e62a0e15a","Type":"ContainerStarted","Data":"0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a"}
Oct 31 20:05:39.300075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:39.299894    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" event={"ID":"d71847a6-27eb-4639-abba-6c9e62a0e15a","Type":"ContainerStarted","Data":"9d7ba09d5e01eecb2c09a0a7136026665ee57931b589553baade1d86bbf8ce15"}
Oct 31 20:05:39.306211 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:39.306184    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:39.307882 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:39.307853    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gjb7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 20:05:39.308022 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:39.307908    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 20:05:39.420508 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:39.420442    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" podStartSLOduration=2.420423203 podStartE2EDuration="2.420423203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:39.416726143 +0000 UTC m=+2957.946260196" watchObservedRunningTime="2026-10-31 20:05:39.420423203 +0000 UTC m=+2957.949957257"
Oct 31 20:05:39.623945 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:39.623892    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847f87ee-2302-423f-a9f8-5a307a2e7b1a" path="/var/lib/kubelet/pods/847f87ee-2302-423f-a9f8-5a307a2e7b1a/volumes"
Oct 31 20:05:39.624617 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:39.624591    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c7c3d9-eda3-432e-91b6-40496d56f48e" path="/var/lib/kubelet/pods/b5c7c3d9-eda3-432e-91b6-40496d56f48e/volumes"
Oct 31 20:05:40.021597 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:40.021500    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:40.313275 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:40.313172    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gjb7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 20:05:40.313587 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:40.313534    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 20:05:41.048025 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:41.047985    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"]
Oct 31 20:05:41.117677 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:41.113052    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e255ed5e-7shsr"]
Oct 31 20:05:41.320911 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:41.320817    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gjb7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 20:05:41.320911 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:41.320878    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 20:05:41.620737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:41.620688    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64204cf0-9632-49e2-8e2d-03612a4d6116" path="/var/lib/kubelet/pods/64204cf0-9632-49e2-8e2d-03612a4d6116/volumes"
Oct 31 20:05:42.318709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:42.318108    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gjb7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 20:05:42.318709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:42.318174    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 20:05:42.618678 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:42.615882    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:43.321137 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:43.321088    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gjb7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 20:05:43.321572 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:43.321161    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 20:05:43.465329 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:43.465292    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:44.309676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:44.307767    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gjb7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 20:05:44.309676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:44.307840    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 20:05:45.306380 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:45.306336    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gjb7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 20:05:45.306846 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:45.306407    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 20:05:46.306854 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:46.306807    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gjb7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 31 20:05:46.307290 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:46.306896    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 31 20:05:46.898225 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:46.898185    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:46.936540 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:46.936490    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9" podUID="c96b94c3-1d84-46a1-8898-d9a32e19816f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:47.647932 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.647897    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:47.648415 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.648302    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:47.649296 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.649274    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:05:47.757437 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.757399    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"]
Oct 31 20:05:47.768750 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.768638    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:47.771375 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.771323    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"]
Oct 31 20:05:47.776463 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.776436    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"]
Oct 31 20:05:47.776598 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.776562    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:47.786372 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.786342    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5776c90d-wcfsd\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:47.786574 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.786552    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf4w8\" (UniqueName: \"kubernetes.io/projected/137496ba-aa62-42b7-92a4-e802faf99afc-kube-api-access-xf4w8\") pod \"maas-api-revoke-sub-5776c90d-wcfsd\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:47.786707 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.786599    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5776c90d-wcfsd\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:47.821210 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.821174    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:05:47.821374 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.821236    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:05:47.821374 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.821264    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:05:47.861953 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.861915    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"]
Oct 31 20:05:47.888359 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.888323    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5776c90d-wcfsd\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:47.888576 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.888375    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f7d4cb1-52gz2\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:47.888576 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.888426    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxm8j\" (UniqueName: \"kubernetes.io/projected/efe0d09d-180f-40fd-ad90-21a65f5245c8-kube-api-access-nxm8j\") pod \"maas-api-revoke-sub-0f7d4cb1-52gz2\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:47.888762 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.888573    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf4w8\" (UniqueName: \"kubernetes.io/projected/137496ba-aa62-42b7-92a4-e802faf99afc-kube-api-access-xf4w8\") pod \"maas-api-revoke-sub-5776c90d-wcfsd\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:47.888762 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.888621    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f7d4cb1-52gz2\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:47.888762 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.888645    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5776c90d-wcfsd\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:47.889140 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.889116    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5776c90d-wcfsd\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:47.891291 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.891271    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5776c90d-wcfsd\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:47.921613 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.921522    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:05:47.945685 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.945638    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf4w8\" (UniqueName: \"kubernetes.io/projected/137496ba-aa62-42b7-92a4-e802faf99afc-kube-api-access-xf4w8\") pod \"maas-api-revoke-sub-5776c90d-wcfsd\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:47.990498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.990451    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxm8j\" (UniqueName: \"kubernetes.io/projected/efe0d09d-180f-40fd-ad90-21a65f5245c8-kube-api-access-nxm8j\") pod \"maas-api-revoke-sub-0f7d4cb1-52gz2\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:47.990775 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.990685    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f7d4cb1-52gz2\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:47.991544 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.991517    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f7d4cb1-52gz2\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:47.992504 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.992479    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f7d4cb1-52gz2\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:47.994484 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:47.994460    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f7d4cb1-52gz2\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:48.052918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:48.052884    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxm8j\" (UniqueName: \"kubernetes.io/projected/efe0d09d-180f-40fd-ad90-21a65f5245c8-kube-api-access-nxm8j\") pod \"maas-api-revoke-sub-0f7d4cb1-52gz2\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:48.101852 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:48.101819    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:48.105964 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:48.105935    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:48.525778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:48.524384    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"]
Oct 31 20:05:48.563199 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:05:48.563169    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe0d09d_180f_40fd_ad90_21a65f5245c8.slice/crio-3b97f0bb25ac6d8606d1cd7c2f5ee7b818098a55a04eaecff51feb03cd82d0ea WatchSource:0}: Error finding container 3b97f0bb25ac6d8606d1cd7c2f5ee7b818098a55a04eaecff51feb03cd82d0ea: Status 404 returned error can't find the container with id 3b97f0bb25ac6d8606d1cd7c2f5ee7b818098a55a04eaecff51feb03cd82d0ea
Oct 31 20:05:48.566581 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:48.564841    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"]
Oct 31 20:05:48.781054 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:48.780972    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:48.802598 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:48.802096    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:05:49.402794 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:49.402754    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd" event={"ID":"137496ba-aa62-42b7-92a4-e802faf99afc","Type":"ContainerStarted","Data":"d29baf2059486f2c53a58ad54766d561579c5ce6fafc4e6f983bd9c63cbcd1ca"}
Oct 31 20:05:49.402958 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:49.402817    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd" event={"ID":"137496ba-aa62-42b7-92a4-e802faf99afc","Type":"ContainerStarted","Data":"ae7cff2bda773d458d5012d891e151ac9ef8635ae03ec387fae534c8ddbf8cf6"}
Oct 31 20:05:49.404783 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:49.404754    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2" event={"ID":"efe0d09d-180f-40fd-ad90-21a65f5245c8","Type":"ContainerStarted","Data":"4f35a1edc65de2f2e61198ef7cf8b9f51c4eb10079b870a701f09fd05da1d85a"}
Oct 31 20:05:49.404926 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:49.404801    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2" event={"ID":"efe0d09d-180f-40fd-ad90-21a65f5245c8","Type":"ContainerStarted","Data":"3b97f0bb25ac6d8606d1cd7c2f5ee7b818098a55a04eaecff51feb03cd82d0ea"}
Oct 31 20:05:49.522705 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:49.522637    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd" podStartSLOduration=2.522622889 podStartE2EDuration="2.522622889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:49.521180094 +0000 UTC m=+2968.050714146" watchObservedRunningTime="2026-10-31 20:05:49.522622889 +0000 UTC m=+2968.052156942"
Oct 31 20:05:49.640145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:49.640087    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2" podStartSLOduration=2.640068627 podStartE2EDuration="2.640068627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:49.636928585 +0000 UTC m=+2968.166462638" watchObservedRunningTime="2026-10-31 20:05:49.640068627 +0000 UTC m=+2968.169602681"
Oct 31 20:05:49.739214 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:05:49.739178    5006 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137496ba_aa62_42b7_92a4_e802faf99afc.slice/crio-conmon-d29baf2059486f2c53a58ad54766d561579c5ce6fafc4e6f983bd9c63cbcd1ca.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 20:05:50.415825 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:50.415781    5006 generic.go:356] "Generic (PLEG): container finished" podID="efe0d09d-180f-40fd-ad90-21a65f5245c8" containerID="4f35a1edc65de2f2e61198ef7cf8b9f51c4eb10079b870a701f09fd05da1d85a" exitCode=0
Oct 31 20:05:50.416363 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:50.415823    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2" event={"ID":"efe0d09d-180f-40fd-ad90-21a65f5245c8","Type":"ContainerDied","Data":"4f35a1edc65de2f2e61198ef7cf8b9f51c4eb10079b870a701f09fd05da1d85a"}
Oct 31 20:05:50.417776 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:50.417744    5006 generic.go:356] "Generic (PLEG): container finished" podID="137496ba-aa62-42b7-92a4-e802faf99afc" containerID="d29baf2059486f2c53a58ad54766d561579c5ce6fafc4e6f983bd9c63cbcd1ca" exitCode=0
Oct 31 20:05:50.417927 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:50.417833    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd" event={"ID":"137496ba-aa62-42b7-92a4-e802faf99afc","Type":"ContainerDied","Data":"d29baf2059486f2c53a58ad54766d561579c5ce6fafc4e6f983bd9c63cbcd1ca"}
Oct 31 20:05:51.620181 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.619388    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:51.650957 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.650920    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-xks-ca\") pod \"137496ba-aa62-42b7-92a4-e802faf99afc\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") "
Oct 31 20:05:51.651119 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.650927    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:51.651119 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.651029    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-openshift-ca\") pod \"137496ba-aa62-42b7-92a4-e802faf99afc\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") "
Oct 31 20:05:51.651119 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.651082    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137496ba-aa62-42b7-92a4-e802faf99afc-kube-api-access-xf4w8\" (UniqueName: \"kubernetes.io/projected/137496ba-aa62-42b7-92a4-e802faf99afc-kube-api-access-xf4w8\") pod \"137496ba-aa62-42b7-92a4-e802faf99afc\" (UID: \"137496ba-aa62-42b7-92a4-e802faf99afc\") "
Oct 31 20:05:51.651506 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.651463    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-openshift-ca" pod "137496ba-aa62-42b7-92a4-e802faf99afc" (UID: "137496ba-aa62-42b7-92a4-e802faf99afc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:51.651878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.651849    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:51.653136 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.653112    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-xks-ca" pod "137496ba-aa62-42b7-92a4-e802faf99afc" (UID: "137496ba-aa62-42b7-92a4-e802faf99afc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:51.653452 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.653433    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137496ba-aa62-42b7-92a4-e802faf99afc-kube-api-access-xf4w8" pod "137496ba-aa62-42b7-92a4-e802faf99afc" (UID: "137496ba-aa62-42b7-92a4-e802faf99afc"). InnerVolumeSpecName "kube-api-access-xf4w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:51.752693 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.752575    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe0d09d-180f-40fd-ad90-21a65f5245c8-kube-api-access-nxm8j\" (UniqueName: \"kubernetes.io/projected/efe0d09d-180f-40fd-ad90-21a65f5245c8-kube-api-access-nxm8j\") pod \"efe0d09d-180f-40fd-ad90-21a65f5245c8\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") "
Oct 31 20:05:51.752885 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.752748    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-openshift-ca\") pod \"efe0d09d-180f-40fd-ad90-21a65f5245c8\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") "
Oct 31 20:05:51.752962 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.752915    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-xks-ca\") pod \"efe0d09d-180f-40fd-ad90-21a65f5245c8\" (UID: \"efe0d09d-180f-40fd-ad90-21a65f5245c8\") "
Oct 31 20:05:51.753177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.753149    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-openshift-ca" pod "efe0d09d-180f-40fd-ad90-21a65f5245c8" (UID: "efe0d09d-180f-40fd-ad90-21a65f5245c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:51.753464 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.753442    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137496ba-aa62-42b7-92a4-e802faf99afc-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:51.753569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.753470    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf4w8\" (UniqueName: \"kubernetes.io/projected/137496ba-aa62-42b7-92a4-e802faf99afc-kube-api-access-xf4w8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:51.753569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.753485    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:51.755369 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.755346    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-xks-ca" pod "efe0d09d-180f-40fd-ad90-21a65f5245c8" (UID: "efe0d09d-180f-40fd-ad90-21a65f5245c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:51.755615 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.755596    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe0d09d-180f-40fd-ad90-21a65f5245c8-kube-api-access-nxm8j" pod "efe0d09d-180f-40fd-ad90-21a65f5245c8" (UID: "efe0d09d-180f-40fd-ad90-21a65f5245c8"). InnerVolumeSpecName "kube-api-access-nxm8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:51.855062 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.855019    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxm8j\" (UniqueName: \"kubernetes.io/projected/efe0d09d-180f-40fd-ad90-21a65f5245c8-kube-api-access-nxm8j\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:51.855062 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:51.855060    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe0d09d-180f-40fd-ad90-21a65f5245c8-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:52.435472 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:52.435444    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"
Oct 31 20:05:52.435643 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:52.435481    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2" event={"ID":"efe0d09d-180f-40fd-ad90-21a65f5245c8","Type":"ContainerDied","Data":"3b97f0bb25ac6d8606d1cd7c2f5ee7b818098a55a04eaecff51feb03cd82d0ea"}
Oct 31 20:05:52.435643 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:52.435540    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b97f0bb25ac6d8606d1cd7c2f5ee7b818098a55a04eaecff51feb03cd82d0ea"
Oct 31 20:05:52.436925 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:52.436905    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"
Oct 31 20:05:52.437034 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:52.436942    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd" event={"ID":"137496ba-aa62-42b7-92a4-e802faf99afc","Type":"ContainerDied","Data":"ae7cff2bda773d458d5012d891e151ac9ef8635ae03ec387fae534c8ddbf8cf6"}
Oct 31 20:05:52.437034 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:52.436983    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7cff2bda773d458d5012d891e151ac9ef8635ae03ec387fae534c8ddbf8cf6"
Oct 31 20:05:53.807082 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:53.807048    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:56.469707 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:56.469669    5006 generic.go:356] "Generic (PLEG): container finished" podID="c96b94c3-1d84-46a1-8898-d9a32e19816f" containerID="a991994978f70d09c077c14a4ce7513f8b6dcd506d32a9c13ffd52ad2d256c8e" exitCode=0
Oct 31 20:05:56.470171 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:56.469935    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9" event={"ID":"c96b94c3-1d84-46a1-8898-d9a32e19816f","Type":"ContainerDied","Data":"a991994978f70d09c077c14a4ce7513f8b6dcd506d32a9c13ffd52ad2d256c8e"}
Oct 31 20:05:56.499326 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:56.499260    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 20:05:56.620967 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:56.620932    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c96b94c3-1d84-46a1-8898-d9a32e19816f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96b94c3-1d84-46a1-8898-d9a32e19816f-plugins-config-volume\") pod \"c96b94c3-1d84-46a1-8898-d9a32e19816f\" (UID: \"c96b94c3-1d84-46a1-8898-d9a32e19816f\") "
Oct 31 20:05:56.621159 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:56.621141    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c96b94c3-1d84-46a1-8898-d9a32e19816f-kube-api-access-fdzhb\" (UniqueName: \"kubernetes.io/projected/c96b94c3-1d84-46a1-8898-d9a32e19816f-kube-api-access-fdzhb\") pod \"c96b94c3-1d84-46a1-8898-d9a32e19816f\" (UID: \"c96b94c3-1d84-46a1-8898-d9a32e19816f\") "
Oct 31 20:05:56.621384 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:56.621347    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96b94c3-1d84-46a1-8898-d9a32e19816f-plugins-config-volume" pod "c96b94c3-1d84-46a1-8898-d9a32e19816f" (UID: "c96b94c3-1d84-46a1-8898-d9a32e19816f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:56.621638 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:56.621622    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96b94c3-1d84-46a1-8898-d9a32e19816f-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:56.623265 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:56.623237    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96b94c3-1d84-46a1-8898-d9a32e19816f-kube-api-access-fdzhb" pod "c96b94c3-1d84-46a1-8898-d9a32e19816f" (UID: "c96b94c3-1d84-46a1-8898-d9a32e19816f"). InnerVolumeSpecName "kube-api-access-fdzhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:56.722940 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:56.722904    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdzhb\" (UniqueName: \"kubernetes.io/projected/c96b94c3-1d84-46a1-8898-d9a32e19816f-kube-api-access-fdzhb\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:57.478274 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:57.478248    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"
Oct 31 20:05:57.478777 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:57.478284    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9" event={"ID":"c96b94c3-1d84-46a1-8898-d9a32e19816f","Type":"ContainerDied","Data":"2d760cb8ff652c6185e24e2df6fe97795105b0183d74b148f8a117a2874a00a1"}
Oct 31 20:05:57.478777 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:57.478346    5006 scope.go:122] "RemoveContainer" containerID="a991994978f70d09c077c14a4ce7513f8b6dcd506d32a9c13ffd52ad2d256c8e"
Oct 31 20:05:57.645883 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:57.645850    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"]
Oct 31 20:05:57.695878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:57.695840    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-570915-65fcdf7947-hncp9"]
Oct 31 20:05:58.328615 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:58.328575    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:58.560111 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:58.560071    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:58.973855 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:58.973814    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:59.599276 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:59.599242    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c96b94c3-1d84-46a1-8898-d9a32e19816f" path="/var/lib/kubelet/pods/c96b94c3-1d84-46a1-8898-d9a32e19816f/volumes"
Oct 31 20:05:59.938007 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:05:59.937919    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:00.352920 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:00.352887    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:00.597329 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:00.597288    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:00.885472 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:00.885412    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:03.567295 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:03.567262    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:05.394470 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:06:05.394430    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:06:05.653105 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:05.653022    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:06.651666 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:06.651629    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:08.376939 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:08.376871    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:13.224201 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:13.224168    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:14.941769 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:14.941719    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:15.589833 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:15.589794    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:18.081087 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:18.081036    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:22.197564 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:22.197523    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:06:22.278177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:22.278150    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:06:22.566572 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:22.566491    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:22.924798 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:22.924712    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:23.138531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:23.138489    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:24.297828 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:24.297791    5006 scope.go:122] "RemoveContainer" containerID="23ac4c086611a6f69c99f2b0abcc8564b2d00bff6037a15226ce6f846531ba12"
Oct 31 20:06:24.309234 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:24.309207    5006 scope.go:122] "RemoveContainer" containerID="0e08fc04be9d6050716999e82d8e0d4ee353a85ecd80d5f624956761fa7fd97f"
Oct 31 20:06:24.320173 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:24.320145    5006 scope.go:122] "RemoveContainer" containerID="6b552e68eda5a1d5a50ec33ae41dd669030ba511dd69ef4ea0c32fb706907636"
Oct 31 20:06:27.422405 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:27.422362    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:28.583486 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:28.583453    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:29.036451 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:29.036419    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:29.290457 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:29.290282    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:29.464351 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:29.464311    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"]
Oct 31 20:06:29.510525 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:29.510486    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:29.514738 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:29.514699    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b5155c-p4l6l"]
Oct 31 20:06:29.599191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:29.599155    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188cace0-24fc-4e33-a02f-53e71317cbe0" path="/var/lib/kubelet/pods/188cace0-24fc-4e33-a02f-53e71317cbe0/volumes"
Oct 31 20:06:29.965077 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:29.964900    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:32.297156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:32.294536    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:32.511268 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:32.511225    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:33.178107 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.178072    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:33.584608 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.584521    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"]
Oct 31 20:06:33.585156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.585141    5006 cpu_manager.go:414] "containerMap: removing container" podUID="efe0d09d-180f-40fd-ad90-21a65f5245c8" containerName="revoke-keys"
Oct 31 20:06:33.585156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.585157    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe0d09d-180f-40fd-ad90-21a65f5245c8" containerName="revoke-keys"
Oct 31 20:06:33.585236 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.585193    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c96b94c3-1d84-46a1-8898-d9a32e19816f" containerName="payload-processing"
Oct 31 20:06:33.585236 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.585199    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96b94c3-1d84-46a1-8898-d9a32e19816f" containerName="payload-processing"
Oct 31 20:06:33.585236 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.585212    5006 cpu_manager.go:414] "containerMap: removing container" podUID="137496ba-aa62-42b7-92a4-e802faf99afc" containerName="revoke-keys"
Oct 31 20:06:33.585236 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.585217    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137496ba-aa62-42b7-92a4-e802faf99afc" containerName="revoke-keys"
Oct 31 20:06:33.585363 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.585323    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96b94c3-1d84-46a1-8898-d9a32e19816f" containerName="payload-processing"
Oct 31 20:06:33.585363 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.585333    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="137496ba-aa62-42b7-92a4-e802faf99afc" containerName="revoke-keys"
Oct 31 20:06:33.585363 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.585343    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe0d09d-180f-40fd-ad90-21a65f5245c8" containerName="revoke-keys"
Oct 31 20:06:33.591191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.591152    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:33.622838 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.622795    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68vl\" (UniqueName: \"kubernetes.io/projected/feac09ea-713a-4e42-ba72-7ac3ff003cf0-kube-api-access-d68vl\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:33.622838 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.622843    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:33.623072 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.622972    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:33.643728 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.643691    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"]
Oct 31 20:06:33.689322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.689264    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:06:33.689569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.689412    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:06:33.689569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.689466    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:06:33.694290 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.694265    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:06:33.723893 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.723862    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68vl\" (UniqueName: \"kubernetes.io/projected/feac09ea-713a-4e42-ba72-7ac3ff003cf0-kube-api-access-d68vl\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:33.724068 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.723903    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:33.724068 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.723961    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:33.724907 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.724880    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:33.726488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.726463    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:33.782242 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.782200    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68vl\" (UniqueName: \"kubernetes.io/projected/feac09ea-713a-4e42-ba72-7ac3ff003cf0-kube-api-access-d68vl\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:33.907122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:33.907085    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:34.301194 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:34.301155    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"]
Oct 31 20:06:34.544751 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:34.544621    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:34.785182 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:34.785131    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8" event={"ID":"feac09ea-713a-4e42-ba72-7ac3ff003cf0","Type":"ContainerStarted","Data":"325c8f835f51cc1d71c895565cdde865e9170eb41a46330260d2e94c59b5c36b"}
Oct 31 20:06:34.785592 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:34.785199    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8" event={"ID":"feac09ea-713a-4e42-ba72-7ac3ff003cf0","Type":"ContainerStarted","Data":"d401ffa17b5e608500702555a1b9f1e1285b1bf82d92f708047b3fa2ccbe3c34"}
Oct 31 20:06:34.904163 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:34.904089    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8" podStartSLOduration=1.904070189 podStartE2EDuration="1.904070189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:06:34.901609223 +0000 UTC m=+3013.431143276" watchObservedRunningTime="2026-10-31 20:06:34.904070189 +0000 UTC m=+3013.433604242"
Oct 31 20:06:35.395424 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:06:35.395384    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:06:35.796066 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:35.796032    5006 generic.go:356] "Generic (PLEG): container finished" podID="feac09ea-713a-4e42-ba72-7ac3ff003cf0" containerID="325c8f835f51cc1d71c895565cdde865e9170eb41a46330260d2e94c59b5c36b" exitCode=0
Oct 31 20:06:35.796511 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:35.796084    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8" event={"ID":"feac09ea-713a-4e42-ba72-7ac3ff003cf0","Type":"ContainerDied","Data":"325c8f835f51cc1d71c895565cdde865e9170eb41a46330260d2e94c59b5c36b"}
Oct 31 20:06:35.973766 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:35.973714    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:36.234370 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:36.234333    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"]
Oct 31 20:06:36.234852 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:36.234820    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy" containerID="cri-o://18cdf54dcf17a807d7cce0cc3d64f2d7b86b42c4b58b7e15924f6c542e0469a7" gracePeriod=30
Oct 31 20:06:36.235127 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:36.235098    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:36.486435 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:36.486341    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:36.985547 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:36.985512    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:37.075719 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.075581    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feac09ea-713a-4e42-ba72-7ac3ff003cf0-kube-api-access-d68vl\" (UniqueName: \"kubernetes.io/projected/feac09ea-713a-4e42-ba72-7ac3ff003cf0-kube-api-access-d68vl\") pod \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") "
Oct 31 20:06:37.075719 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.075669    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-openshift-ca\") pod \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") "
Oct 31 20:06:37.075719 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.075704    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-xks-ca\") pod \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\" (UID: \"feac09ea-713a-4e42-ba72-7ac3ff003cf0\") "
Oct 31 20:06:37.076085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.076057    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-openshift-ca" pod "feac09ea-713a-4e42-ba72-7ac3ff003cf0" (UID: "feac09ea-713a-4e42-ba72-7ac3ff003cf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:37.076285 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.076267    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:37.077841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.077817    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-xks-ca" pod "feac09ea-713a-4e42-ba72-7ac3ff003cf0" (UID: "feac09ea-713a-4e42-ba72-7ac3ff003cf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:37.077932 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.077897    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feac09ea-713a-4e42-ba72-7ac3ff003cf0-kube-api-access-d68vl" pod "feac09ea-713a-4e42-ba72-7ac3ff003cf0" (UID: "feac09ea-713a-4e42-ba72-7ac3ff003cf0"). InnerVolumeSpecName "kube-api-access-d68vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:37.177750 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.177704    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d68vl\" (UniqueName: \"kubernetes.io/projected/feac09ea-713a-4e42-ba72-7ac3ff003cf0-kube-api-access-d68vl\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:37.177750 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.177741    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feac09ea-713a-4e42-ba72-7ac3ff003cf0-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:37.813186 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.813154    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"
Oct 31 20:06:37.813186 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.813182    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8" event={"ID":"feac09ea-713a-4e42-ba72-7ac3ff003cf0","Type":"ContainerDied","Data":"d401ffa17b5e608500702555a1b9f1e1285b1bf82d92f708047b3fa2ccbe3c34"}
Oct 31 20:06:37.813413 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:37.813227    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d401ffa17b5e608500702555a1b9f1e1285b1bf82d92f708047b3fa2ccbe3c34"
Oct 31 20:06:38.049902 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:38.049811    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"]
Oct 31 20:06:38.067075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:38.067042    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"]
Oct 31 20:06:38.099452 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:38.099417    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"]
Oct 31 20:06:38.306305 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:38.306213    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:39.804493 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:39.804454    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:39.819376 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:39.819341    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:06:41.847573 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:41.847540    5006 generic.go:356] "Generic (PLEG): container finished" podID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerID="18cdf54dcf17a807d7cce0cc3d64f2d7b86b42c4b58b7e15924f6c542e0469a7" exitCode=0
Oct 31 20:06:41.847987 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:41.847664    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" event={"ID":"f4ea5406-4c0a-4faf-8e91-230b2e442e63","Type":"ContainerDied","Data":"18cdf54dcf17a807d7cce0cc3d64f2d7b86b42c4b58b7e15924f6c542e0469a7"}
Oct 31 20:06:41.902832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:41.902768    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:41.964100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:41.964072    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 20:06:42.054216 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054113    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-credential-socket\") pod \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") "
Oct 31 20:06:42.054216 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054196    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-socket\") pod \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") "
Oct 31 20:06:42.054493 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054238    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istiod-ca-cert\") pod \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") "
Oct 31 20:06:42.054493 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054347    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-data\") pod \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") "
Oct 31 20:06:42.054493 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054417    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-certs\") pod \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") "
Oct 31 20:06:42.054493 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054447    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-credential-socket" pod "f4ea5406-4c0a-4faf-8e91-230b2e442e63" (UID: "f4ea5406-4c0a-4faf-8e91-230b2e442e63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:42.054493 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054465    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-token\" (UniqueName: \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-token\") pod \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") "
Oct 31 20:06:42.054809 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054504    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-kube-api-access-mrlw2\" (UniqueName: \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-kube-api-access-mrlw2\") pod \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") "
Oct 31 20:06:42.054809 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054546    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-envoy\") pod \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") "
Oct 31 20:06:42.054809 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054562    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-socket" pod "f4ea5406-4c0a-4faf-8e91-230b2e442e63" (UID: "f4ea5406-4c0a-4faf-8e91-230b2e442e63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:42.054809 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054573    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-podinfo\") pod \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\" (UID: \"f4ea5406-4c0a-4faf-8e91-230b2e442e63\") "
Oct 31 20:06:42.054809 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054766    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-data" pod "f4ea5406-4c0a-4faf-8e91-230b2e442e63" (UID: "f4ea5406-4c0a-4faf-8e91-230b2e442e63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:42.055090 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.054799    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istiod-ca-cert" pod "f4ea5406-4c0a-4faf-8e91-230b2e442e63" (UID: "f4ea5406-4c0a-4faf-8e91-230b2e442e63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:42.055090 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.055038    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-certs" pod "f4ea5406-4c0a-4faf-8e91-230b2e442e63" (UID: "f4ea5406-4c0a-4faf-8e91-230b2e442e63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:42.055233 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.055213    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:42.055293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.055237    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:42.055293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.055253    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:42.055293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.055268    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:42.055293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.055282    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:42.058103 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.058070    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-envoy" pod "f4ea5406-4c0a-4faf-8e91-230b2e442e63" (UID: "f4ea5406-4c0a-4faf-8e91-230b2e442e63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:42.058327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.058116    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-kube-api-access-mrlw2" pod "f4ea5406-4c0a-4faf-8e91-230b2e442e63" (UID: "f4ea5406-4c0a-4faf-8e91-230b2e442e63"). InnerVolumeSpecName "kube-api-access-mrlw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:42.058327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.058185    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-podinfo" pod "f4ea5406-4c0a-4faf-8e91-230b2e442e63" (UID: "f4ea5406-4c0a-4faf-8e91-230b2e442e63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:06:42.058327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.058281    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-token" pod "f4ea5406-4c0a-4faf-8e91-230b2e442e63" (UID: "f4ea5406-4c0a-4faf-8e91-230b2e442e63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:42.156976 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.156932    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:42.156976 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.156981    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrlw2\" (UniqueName: \"kubernetes.io/projected/f4ea5406-4c0a-4faf-8e91-230b2e442e63-kube-api-access-mrlw2\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:42.157278 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.156999    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:42.157278 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.157014    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4ea5406-4c0a-4faf-8e91-230b2e442e63-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:42.855970 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.855931    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"
Oct 31 20:06:42.855970 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.855957    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz" event={"ID":"f4ea5406-4c0a-4faf-8e91-230b2e442e63","Type":"ContainerDied","Data":"0141675fd0aca0658c4a9704e122b1392be2436f2b31c61c11368bc47fe64dea"}
Oct 31 20:06:42.856475 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:42.856020    5006 scope.go:122] "RemoveContainer" containerID="18cdf54dcf17a807d7cce0cc3d64f2d7b86b42c4b58b7e15924f6c542e0469a7"
Oct 31 20:06:43.074449 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:43.074414    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"]
Oct 31 20:06:43.123845 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:43.123761    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-8jzhz"]
Oct 31 20:06:43.599581 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:43.599544    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" path="/var/lib/kubelet/pods/f4ea5406-4c0a-4faf-8e91-230b2e442e63/volumes"
Oct 31 20:06:44.534084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:44.534032    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:44.756803 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:44.756767    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:44.977563 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:44.977515    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:47.316613 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:47.316577    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:49.189589 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:49.189549    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:50.756841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:50.756807    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:50.984984 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:50.984949    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:51.204559 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:51.204523    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"]
Oct 31 20:06:51.210948 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:51.210921    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:51.254960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:51.254919    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-570915-cx6xv"]
Oct 31 20:06:51.599375 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:51.599337    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f958db3d-0ef2-46a0-9126-eef121f5e577" path="/var/lib/kubelet/pods/f958db3d-0ef2-46a0-9126-eef121f5e577/volumes"
Oct 31 20:06:51.808939 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:51.808901    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:53.072067 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.072020    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main" containerID="cri-o://6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451" gracePeriod=45
Oct 31 20:06:53.088169 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.088111    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" containerID="cri-o://f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d" gracePeriod=45
Oct 31 20:06:53.120622 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.120580    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main" containerID="cri-o://7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b" gracePeriod=45
Oct 31 20:06:53.186118 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.186082    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:53.431558 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.431450    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:53.443488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.443456    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 20:06:53.460831 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.460806    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 20:06:53.466248 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.466216    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 20:06:53.536086 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536049    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-dshm\") pod \"7ee071cd-147d-4bd0-aece-a823c2e7e911\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") "
Oct 31 20:06:53.536299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536098    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-kube-api-access-ldfrr\" (UniqueName: \"kubernetes.io/projected/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-kube-api-access-ldfrr\") pod \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") "
Oct 31 20:06:53.536299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536127    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-dshm\") pod \"240043a0-2aea-451c-8658-d350ba4813e8\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") "
Oct 31 20:06:53.536299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536153    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-model-cache\") pod \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") "
Oct 31 20:06:53.536299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536178    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-home\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-home\") pod \"7ee071cd-147d-4bd0-aece-a823c2e7e911\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") "
Oct 31 20:06:53.536299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536205    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-model-cache\") pod \"7ee071cd-147d-4bd0-aece-a823c2e7e911\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") "
Oct 31 20:06:53.536299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536235    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee071cd-147d-4bd0-aece-a823c2e7e911-kube-api-access-z8zj7\" (UniqueName: \"kubernetes.io/projected/7ee071cd-147d-4bd0-aece-a823c2e7e911-kube-api-access-z8zj7\") pod \"7ee071cd-147d-4bd0-aece-a823c2e7e911\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") "
Oct 31 20:06:53.536759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536349    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ee071cd-147d-4bd0-aece-a823c2e7e911-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee071cd-147d-4bd0-aece-a823c2e7e911-tls-certs\") pod \"7ee071cd-147d-4bd0-aece-a823c2e7e911\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") "
Oct 31 20:06:53.536759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536393    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tls-certs\") pod \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") "
Oct 31 20:06:53.536759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536431    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-model-cache" pod "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" (UID: "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.536759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536438    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-model-cache\") pod \"240043a0-2aea-451c-8658-d350ba4813e8\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") "
Oct 31 20:06:53.536759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536569    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240043a0-2aea-451c-8658-d350ba4813e8-kube-api-access-lsrbl\" (UniqueName: \"kubernetes.io/projected/240043a0-2aea-451c-8658-d350ba4813e8-kube-api-access-lsrbl\") pod \"240043a0-2aea-451c-8658-d350ba4813e8\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") "
Oct 31 20:06:53.536759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536619    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-home\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-home\") pod \"240043a0-2aea-451c-8658-d350ba4813e8\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") "
Oct 31 20:06:53.536759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536624    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-model-cache" pod "240043a0-2aea-451c-8658-d350ba4813e8" (UID: "240043a0-2aea-451c-8658-d350ba4813e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.536759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536673    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-tmp-dir\") pod \"7ee071cd-147d-4bd0-aece-a823c2e7e911\" (UID: \"7ee071cd-147d-4bd0-aece-a823c2e7e911\") "
Oct 31 20:06:53.536759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536706    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-dshm\") pod \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") "
Oct 31 20:06:53.536759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536734    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-home\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-home\") pod \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") "
Oct 31 20:06:53.537322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536798    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/240043a0-2aea-451c-8658-d350ba4813e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/240043a0-2aea-451c-8658-d350ba4813e8-tls-certs\") pod \"240043a0-2aea-451c-8658-d350ba4813e8\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") "
Oct 31 20:06:53.537322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536868    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tmp-dir\") pod \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\" (UID: \"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806\") "
Oct 31 20:06:53.537322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536933    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-tmp-dir\") pod \"240043a0-2aea-451c-8658-d350ba4813e8\" (UID: \"240043a0-2aea-451c-8658-d350ba4813e8\") "
Oct 31 20:06:53.537322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.536947    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-model-cache" pod "7ee071cd-147d-4bd0-aece-a823c2e7e911" (UID: "7ee071cd-147d-4bd0-aece-a823c2e7e911"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.537322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.537234    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-home" pod "7ee071cd-147d-4bd0-aece-a823c2e7e911" (UID: "7ee071cd-147d-4bd0-aece-a823c2e7e911"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.537675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.537491    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-tmp-dir" pod "240043a0-2aea-451c-8658-d350ba4813e8" (UID: "240043a0-2aea-451c-8658-d350ba4813e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.537750 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.537708    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.537750 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.537730    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.537750 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.537745    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.537915 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.537760    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.537915 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.537773    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.538022 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.537917    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-home" pod "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" (UID: "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.538411 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.538372    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tmp-dir" pod "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" (UID: "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.538569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.538548    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-tmp-dir" pod "7ee071cd-147d-4bd0-aece-a823c2e7e911" (UID: "7ee071cd-147d-4bd0-aece-a823c2e7e911"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.538642 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.538617    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-home" pod "240043a0-2aea-451c-8658-d350ba4813e8" (UID: "240043a0-2aea-451c-8658-d350ba4813e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.539329 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.539303    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-dshm" pod "240043a0-2aea-451c-8658-d350ba4813e8" (UID: "240043a0-2aea-451c-8658-d350ba4813e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.539746 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.539717    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee071cd-147d-4bd0-aece-a823c2e7e911-tls-certs" pod "7ee071cd-147d-4bd0-aece-a823c2e7e911" (UID: "7ee071cd-147d-4bd0-aece-a823c2e7e911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:53.540074 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.540049    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-dshm" pod "7ee071cd-147d-4bd0-aece-a823c2e7e911" (UID: "7ee071cd-147d-4bd0-aece-a823c2e7e911"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.540414 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.540385    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/240043a0-2aea-451c-8658-d350ba4813e8-tls-certs" pod "240043a0-2aea-451c-8658-d350ba4813e8" (UID: "240043a0-2aea-451c-8658-d350ba4813e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:53.540825 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.540799    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-kube-api-access-ldfrr" pod "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" (UID: "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806"). InnerVolumeSpecName "kube-api-access-ldfrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:53.541068 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.541048    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240043a0-2aea-451c-8658-d350ba4813e8-kube-api-access-lsrbl" pod "240043a0-2aea-451c-8658-d350ba4813e8" (UID: "240043a0-2aea-451c-8658-d350ba4813e8"). InnerVolumeSpecName "kube-api-access-lsrbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:53.541240 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.541220    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tls-certs" pod "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" (UID: "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:53.541698 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.541676    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee071cd-147d-4bd0-aece-a823c2e7e911-kube-api-access-z8zj7" pod "7ee071cd-147d-4bd0-aece-a823c2e7e911" (UID: "7ee071cd-147d-4bd0-aece-a823c2e7e911"). InnerVolumeSpecName "kube-api-access-z8zj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:53.541968 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.541943    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-dshm" pod "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" (UID: "fb9ff14e-bcc8-44a2-9a4b-8691f58ce806"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644786    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee071cd-147d-4bd0-aece-a823c2e7e911-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644820    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644835    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsrbl\" (UniqueName: \"kubernetes.io/projected/240043a0-2aea-451c-8658-d350ba4813e8-kube-api-access-lsrbl\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644849    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644863    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644871    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644878    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644885    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240043a0-2aea-451c-8658-d350ba4813e8-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644894    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644902    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee071cd-147d-4bd0-aece-a823c2e7e911-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644910    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldfrr\" (UniqueName: \"kubernetes.io/projected/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806-kube-api-access-ldfrr\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644919    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/240043a0-2aea-451c-8658-d350ba4813e8-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.645398 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.644929    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8zj7\" (UniqueName: \"kubernetes.io/projected/7ee071cd-147d-4bd0-aece-a823c2e7e911-kube-api-access-z8zj7\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.648539 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.648505    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"]
Oct 31 20:06:53.648807 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.648770    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy" containerID="cri-o://a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313" gracePeriod=30
Oct 31 20:06:53.666679 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.665999    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:53.927285 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.927165    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:53.945556 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.945495    5006 generic.go:356] "Generic (PLEG): container finished" podID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerID="6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451" exitCode=0
Oct 31 20:06:53.945794 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.945770    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"
Oct 31 20:06:53.946451 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.946430    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" event={"ID":"7ee071cd-147d-4bd0-aece-a823c2e7e911","Type":"ContainerDied","Data":"6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451"}
Oct 31 20:06:53.946515 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.946471    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49" event={"ID":"7ee071cd-147d-4bd0-aece-a823c2e7e911","Type":"ContainerDied","Data":"c694b7059569dddc3e8aa47ddb4d08c9784cde7f03cfe04d018df00f39e9354c"}
Oct 31 20:06:53.946515 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.946495    5006 scope.go:122] "RemoveContainer" containerID="6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451"
Oct 31 20:06:53.948698 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.948633    5006 generic.go:356] "Generic (PLEG): container finished" podID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerID="7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b" exitCode=0
Oct 31 20:06:53.949122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.948812    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" event={"ID":"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806","Type":"ContainerDied","Data":"7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b"}
Oct 31 20:06:53.949122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.948848    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt" event={"ID":"fb9ff14e-bcc8-44a2-9a4b-8691f58ce806","Type":"ContainerDied","Data":"0befff9483fc7e5e85fabbc5f93447190b4e62e2f1f0102c0c9061a8afffea1c"}
Oct 31 20:06:53.949122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.948823    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"
Oct 31 20:06:53.950490 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.950464    5006 generic.go:356] "Generic (PLEG): container finished" podID="240043a0-2aea-451c-8658-d350ba4813e8" containerID="f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d" exitCode=0
Oct 31 20:06:53.950609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.950541    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" event={"ID":"240043a0-2aea-451c-8658-d350ba4813e8","Type":"ContainerDied","Data":"f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d"}
Oct 31 20:06:53.950609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.950571    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" event={"ID":"240043a0-2aea-451c-8658-d350ba4813e8","Type":"ContainerDied","Data":"c506b6f572f4c8ce752bd917efde286aa0c4749563d3f6838055d623fa5b538f"}
Oct 31 20:06:53.950742 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.950629    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"
Oct 31 20:06:53.958725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.958704    5006 scope.go:122] "RemoveContainer" containerID="6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451"
Oct 31 20:06:53.959053 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:06:53.959035    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451\": container with ID starting with 6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451 not found: ID does not exist" containerID="6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451"
Oct 31 20:06:53.959102 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.959063    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451"} err="failed to get container status \"6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451\": rpc error: code = NotFound desc = could not find container \"6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451\": container with ID starting with 6056e0c9617e1f32e22613b96927bd7acd1412d0cdd451a31501a80f1c281451 not found: ID does not exist"
Oct 31 20:06:53.959102 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.959080    5006 scope.go:122] "RemoveContainer" containerID="7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b"
Oct 31 20:06:53.968495 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.968471    5006 scope.go:122] "RemoveContainer" containerID="7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b"
Oct 31 20:06:53.968860 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:06:53.968831    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b\": container with ID starting with 7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b not found: ID does not exist" containerID="7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b"
Oct 31 20:06:53.968950 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.968872    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b"} err="failed to get container status \"7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b\": rpc error: code = NotFound desc = could not find container \"7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b\": container with ID starting with 7b6120a81d1f94ec738bcd9c332bcabfc39660ebcea17d256d8bdceeb26fc81b not found: ID does not exist"
Oct 31 20:06:53.968950 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.968899    5006 scope.go:122] "RemoveContainer" containerID="f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d"
Oct 31 20:06:53.978062 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.978042    5006 scope.go:122] "RemoveContainer" containerID="f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d"
Oct 31 20:06:53.978352 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:06:53.978333    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d\": container with ID starting with f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d not found: ID does not exist" containerID="f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d"
Oct 31 20:06:53.978408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:53.978360    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d"} err="failed to get container status \"f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d\": rpc error: code = NotFound desc = could not find container \"f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d\": container with ID starting with f06c4221d0d064b4b0b1f064dc3ce4f4b23c1730394ba064b70b8f14b301694d not found: ID does not exist"
Oct 31 20:06:54.118576 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:54.118543    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"]
Oct 31 20:06:54.123253 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:54.123228    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-2-w6-5a4438-kserve-5d4d66f857-wwb49"]
Oct 31 20:06:54.233113 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:54.233029    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"]
Oct 31 20:06:54.236304 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:54.236275    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt"]
Oct 31 20:06:54.243576 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:54.243547    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:54.345689 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:54.345639    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"]
Oct 31 20:06:54.349104 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:54.349076    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/simulator-w6-5a4438-kserve-79b857bb6-9kbqt"]
Oct 31 20:06:54.355843 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:54.355797    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/premium-w6-5a4438-kserve-7977db599-2zbxt" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:06:54.473808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:54.473771    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:55.600365 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:55.600317    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240043a0-2aea-451c-8658-d350ba4813e8" path="/var/lib/kubelet/pods/240043a0-2aea-451c-8658-d350ba4813e8/volumes"
Oct 31 20:06:55.600900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:55.600820    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" path="/var/lib/kubelet/pods/7ee071cd-147d-4bd0-aece-a823c2e7e911/volumes"
Oct 31 20:06:55.601243 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:55.601224    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" path="/var/lib/kubelet/pods/fb9ff14e-bcc8-44a2-9a4b-8691f58ce806/volumes"
Oct 31 20:06:56.422606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:56.422558    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:58.157007 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.156972    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:58.387920 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.387884    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:58.403550 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.403512    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"]
Oct 31 20:06:58.403767 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.403744    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy" containerID="cri-o://cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6" gracePeriod=30
Oct 31 20:06:58.615977 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.615942    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:58.920913 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.920886    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 20:06:58.992982 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.992946    5006 generic.go:356] "Generic (PLEG): container finished" podID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerID="a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313" exitCode=0
Oct 31 20:06:58.993154 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.992995    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"
Oct 31 20:06:58.993154 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.993031    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" event={"ID":"2c0617b1-6e0f-4f50-8634-00d4af4d0133","Type":"ContainerDied","Data":"a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313"}
Oct 31 20:06:58.993154 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.993080    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5" event={"ID":"2c0617b1-6e0f-4f50-8634-00d4af4d0133","Type":"ContainerDied","Data":"11e2a2f039a0b6410832c5285ebe956ed298d64563d7a0de76a93adce96ec49f"}
Oct 31 20:06:58.993154 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:58.993104    5006 scope.go:122] "RemoveContainer" containerID="a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313"
Oct 31 20:06:59.004711 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.004685    5006 scope.go:122] "RemoveContainer" containerID="9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138"
Oct 31 20:06:59.015030 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.015005    5006 scope.go:122] "RemoveContainer" containerID="a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313"
Oct 31 20:06:59.015433 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:06:59.015410    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313\": container with ID starting with a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313 not found: ID does not exist" containerID="a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313"
Oct 31 20:06:59.015552 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.015447    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313"} err="failed to get container status \"a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313\": rpc error: code = NotFound desc = could not find container \"a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313\": container with ID starting with a7a2cbcfd8f632e463592b43a887c3a5726431b4383579b09850753744ef6313 not found: ID does not exist"
Oct 31 20:06:59.015552 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.015472    5006 scope.go:122] "RemoveContainer" containerID="9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138"
Oct 31 20:06:59.015808 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:06:59.015787    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138\": container with ID starting with 9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138 not found: ID does not exist" containerID="9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138"
Oct 31 20:06:59.015867 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.015825    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138"} err="failed to get container status \"9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138\": rpc error: code = NotFound desc = could not find container \"9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138\": container with ID starting with 9ea9820fc5edbf922c114107bc8335561030055968c0d656b2d1572bc66df138 not found: ID does not exist"
Oct 31 20:06:59.019269 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019244    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-socket\") pod \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") "
Oct 31 20:06:59.019409 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019287    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-kube-api-access-gcc8g\" (UniqueName: \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-kube-api-access-gcc8g\") pod \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") "
Oct 31 20:06:59.019409 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019328    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istiod-ca-cert\") pod \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") "
Oct 31 20:06:59.019409 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019397    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-data\") pod \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") "
Oct 31 20:06:59.019583 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019452    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-certs\") pod \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") "
Oct 31 20:06:59.019583 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019489    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-credential-socket\") pod \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") "
Oct 31 20:06:59.019705 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019621    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-envoy\") pod \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") "
Oct 31 20:06:59.019705 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019642    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-socket" pod "2c0617b1-6e0f-4f50-8634-00d4af4d0133" (UID: "2c0617b1-6e0f-4f50-8634-00d4af4d0133"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:59.019882 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019706    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-token\" (UniqueName: \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-token\") pod \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") "
Oct 31 20:06:59.019882 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019750    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-podinfo\") pod \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\" (UID: \"2c0617b1-6e0f-4f50-8634-00d4af4d0133\") "
Oct 31 20:06:59.020002 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019890    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-data" pod "2c0617b1-6e0f-4f50-8634-00d4af4d0133" (UID: "2c0617b1-6e0f-4f50-8634-00d4af4d0133"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:59.020002 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.019906    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-certs" pod "2c0617b1-6e0f-4f50-8634-00d4af4d0133" (UID: "2c0617b1-6e0f-4f50-8634-00d4af4d0133"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:59.020175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.020145    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-credential-socket" pod "2c0617b1-6e0f-4f50-8634-00d4af4d0133" (UID: "2c0617b1-6e0f-4f50-8634-00d4af4d0133"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:59.020349 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.020323    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.020460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.020352    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.020460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.020367    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.020460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.020381    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.020676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.020545    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istiod-ca-cert" pod "2c0617b1-6e0f-4f50-8634-00d4af4d0133" (UID: "2c0617b1-6e0f-4f50-8634-00d4af4d0133"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:59.021991 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.021966    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-token" pod "2c0617b1-6e0f-4f50-8634-00d4af4d0133" (UID: "2c0617b1-6e0f-4f50-8634-00d4af4d0133"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:59.022114 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.022088    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-kube-api-access-gcc8g" pod "2c0617b1-6e0f-4f50-8634-00d4af4d0133" (UID: "2c0617b1-6e0f-4f50-8634-00d4af4d0133"). InnerVolumeSpecName "kube-api-access-gcc8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:59.022170 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.022110    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-podinfo" pod "2c0617b1-6e0f-4f50-8634-00d4af4d0133" (UID: "2c0617b1-6e0f-4f50-8634-00d4af4d0133"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:06:59.022170 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.022145    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-envoy" pod "2c0617b1-6e0f-4f50-8634-00d4af4d0133" (UID: "2c0617b1-6e0f-4f50-8634-00d4af4d0133"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:59.121295 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.121259    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcc8g\" (UniqueName: \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-kube-api-access-gcc8g\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.121295 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.121292    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.121295 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.121301    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.121559 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.121311    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.121559 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.121323    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0617b1-6e0f-4f50-8634-00d4af4d0133-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.416550 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.416508    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"]
Oct 31 20:06:59.467019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.466982    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9902ffd9-openshift-default-544d4875b9-gb6q5"]
Oct 31 20:06:59.605174 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:06:59.605130    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" path="/var/lib/kubelet/pods/2c0617b1-6e0f-4f50-8634-00d4af4d0133/volumes"
Oct 31 20:07:00.671052 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671013    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"]
Oct 31 20:07:00.671585 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671571    5006 cpu_manager.go:414] "containerMap: removing container" podUID="feac09ea-713a-4e42-ba72-7ac3ff003cf0" containerName="revoke-keys"
Oct 31 20:07:00.671672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671586    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feac09ea-713a-4e42-ba72-7ac3ff003cf0" containerName="revoke-keys"
Oct 31 20:07:00.671672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671609    5006 cpu_manager.go:414] "containerMap: removing container" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main"
Oct 31 20:07:00.671672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671615    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main"
Oct 31 20:07:00.671672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671623    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy"
Oct 31 20:07:00.671672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671629    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy"
Oct 31 20:07:00.671672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671643    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy"
Oct 31 20:07:00.671672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671664    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671677    5006 cpu_manager.go:414] "containerMap: removing container" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671684    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671698    5006 cpu_manager.go:414] "containerMap: removing container" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671708    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671725    5006 cpu_manager.go:414] "containerMap: removing container" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671733    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671854    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="feac09ea-713a-4e42-ba72-7ac3ff003cf0" containerName="revoke-keys"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671864    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9ff14e-bcc8-44a2-9a4b-8691f58ce806" containerName="main"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671870    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671879    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ea5406-4c0a-4faf-8e91-230b2e442e63" containerName="istio-proxy"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671885    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee071cd-147d-4bd0-aece-a823c2e7e911" containerName="main"
Oct 31 20:07:00.671897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.671892    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="240043a0-2aea-451c-8658-d350ba4813e8" containerName="main"
Oct 31 20:07:00.676859 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.676836    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:00.728391 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.728357    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"]
Oct 31 20:07:00.731990 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.731952    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"]
Oct 31 20:07:00.733027 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.733000    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0617b1-6e0f-4f50-8634-00d4af4d0133" containerName="istio-proxy"
Oct 31 20:07:00.737142 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.737116    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:00.740954 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.740922    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c674f1a-vdfkh\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:00.741099 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.741050    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9gnt\" (UniqueName: \"kubernetes.io/projected/b69ba68a-797e-4014-aa92-bd60d877492d-kube-api-access-c9gnt\") pod \"maas-api-revoke-sub-0c674f1a-vdfkh\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:00.741230 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.741210    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c674f1a-vdfkh\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:00.775402 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.775370    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:07:00.777702 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.777645    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:07:00.777702 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.777678    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:07:00.777702 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.777682    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:07:00.828992 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.828957    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"]
Oct 31 20:07:00.842807 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.842770    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c674f1a-vdfkh\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:00.842807 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.842812    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0930954e-z5t8w\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:00.843037 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.842854    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rsnx\" (UniqueName: \"kubernetes.io/projected/72cf565b-5277-48cb-bc3e-aeb9246df687-kube-api-access-6rsnx\") pod \"maas-api-revoke-sub-0930954e-z5t8w\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:00.843105 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.843075    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9gnt\" (UniqueName: \"kubernetes.io/projected/b69ba68a-797e-4014-aa92-bd60d877492d-kube-api-access-c9gnt\") pod \"maas-api-revoke-sub-0c674f1a-vdfkh\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:00.843199 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.843180    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0930954e-z5t8w\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:00.843322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.843302    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c674f1a-vdfkh\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:00.843701 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.843644    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c674f1a-vdfkh\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:00.845803 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.845771    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c674f1a-vdfkh\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:00.900602 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.900561    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9gnt\" (UniqueName: \"kubernetes.io/projected/b69ba68a-797e-4014-aa92-bd60d877492d-kube-api-access-c9gnt\") pod \"maas-api-revoke-sub-0c674f1a-vdfkh\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:00.944853 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.944762    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0930954e-z5t8w\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:00.944853 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.944834    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsnx\" (UniqueName: \"kubernetes.io/projected/72cf565b-5277-48cb-bc3e-aeb9246df687-kube-api-access-6rsnx\") pod \"maas-api-revoke-sub-0930954e-z5t8w\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:00.945088 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.944928    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0930954e-z5t8w\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:00.945569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.945529    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0930954e-z5t8w\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:00.947458 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.947427    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0930954e-z5t8w\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:00.990621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:00.990578    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:01.002951 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:01.002924    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rsnx\" (UniqueName: \"kubernetes.io/projected/72cf565b-5277-48cb-bc3e-aeb9246df687-kube-api-access-6rsnx\") pod \"maas-api-revoke-sub-0930954e-z5t8w\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:01.052134 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:01.052094    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:01.374915 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:01.374884    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"]
Oct 31 20:07:01.461833 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:07:01.461800    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cf565b_5277_48cb_bc3e_aeb9246df687.slice/crio-b1cecd5df8daada95d8bae8b8efc5c399c5f5fbeab56def26e3d48fc2ba615a3 WatchSource:0}: Error finding container b1cecd5df8daada95d8bae8b8efc5c399c5f5fbeab56def26e3d48fc2ba615a3: Status 404 returned error can't find the container with id b1cecd5df8daada95d8bae8b8efc5c399c5f5fbeab56def26e3d48fc2ba615a3
Oct 31 20:07:01.463596 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:01.463573    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"]
Oct 31 20:07:01.592110 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:01.592071    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:02.021701 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:02.020886    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w" event={"ID":"72cf565b-5277-48cb-bc3e-aeb9246df687","Type":"ContainerStarted","Data":"7ddeab1bb2575f93e597bc2b81fbdb1fdf96e3442faaa264ff67ca6d158274bb"}
Oct 31 20:07:02.021701 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:02.020960    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w" event={"ID":"72cf565b-5277-48cb-bc3e-aeb9246df687","Type":"ContainerStarted","Data":"b1cecd5df8daada95d8bae8b8efc5c399c5f5fbeab56def26e3d48fc2ba615a3"}
Oct 31 20:07:02.022606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:02.022573    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh" event={"ID":"b69ba68a-797e-4014-aa92-bd60d877492d","Type":"ContainerStarted","Data":"d693b5ea1bc86b5bd58bb941e547acab1d99c0f3413560513b61705643ec21fc"}
Oct 31 20:07:02.022708 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:02.022630    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh" event={"ID":"b69ba68a-797e-4014-aa92-bd60d877492d","Type":"ContainerStarted","Data":"0d8cadf8264abe30f4aa4cddd3aa2613d6e3a2b42f259aa019e14d9663dd8a84"}
Oct 31 20:07:02.137880 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:02.137821    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w" podStartSLOduration=2.137803526 podStartE2EDuration="2.137803526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:07:02.134688949 +0000 UTC m=+3040.664222999" watchObservedRunningTime="2026-10-31 20:07:02.137803526 +0000 UTC m=+3040.667337578"
Oct 31 20:07:02.250615 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:02.250558    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh" podStartSLOduration=2.250541997 podStartE2EDuration="2.250541997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:07:02.24853666 +0000 UTC m=+3040.778070744" watchObservedRunningTime="2026-10-31 20:07:02.250541997 +0000 UTC m=+3040.780076087"
Oct 31 20:07:03.031725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.031690    5006 generic.go:356] "Generic (PLEG): container finished" podID="72cf565b-5277-48cb-bc3e-aeb9246df687" containerID="7ddeab1bb2575f93e597bc2b81fbdb1fdf96e3442faaa264ff67ca6d158274bb" exitCode=0
Oct 31 20:07:03.032257 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.031729    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w" event={"ID":"72cf565b-5277-48cb-bc3e-aeb9246df687","Type":"ContainerDied","Data":"7ddeab1bb2575f93e597bc2b81fbdb1fdf96e3442faaa264ff67ca6d158274bb"}
Oct 31 20:07:03.033420 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.033386    5006 generic.go:356] "Generic (PLEG): container finished" podID="b69ba68a-797e-4014-aa92-bd60d877492d" containerID="d693b5ea1bc86b5bd58bb941e547acab1d99c0f3413560513b61705643ec21fc" exitCode=0
Oct 31 20:07:03.033578 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.033439    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh" event={"ID":"b69ba68a-797e-4014-aa92-bd60d877492d","Type":"ContainerDied","Data":"d693b5ea1bc86b5bd58bb941e547acab1d99c0f3413560513b61705643ec21fc"}
Oct 31 20:07:03.681042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.680974    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 20:07:03.782645 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.782611    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-kube-api-access-x29t8\" (UniqueName: \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-kube-api-access-x29t8\") pod \"c660d7cd-b579-4885-a115-bee9024c128c\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") "
Oct 31 20:07:03.782852 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.782742    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-credential-socket\") pod \"c660d7cd-b579-4885-a115-bee9024c128c\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") "
Oct 31 20:07:03.782852 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.782780    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-certs\") pod \"c660d7cd-b579-4885-a115-bee9024c128c\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") "
Oct 31 20:07:03.782852 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.782841    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-socket\") pod \"c660d7cd-b579-4885-a115-bee9024c128c\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") "
Oct 31 20:07:03.782952 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.782940    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-envoy\") pod \"c660d7cd-b579-4885-a115-bee9024c128c\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") "
Oct 31 20:07:03.783008 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.782989    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c660d7cd-b579-4885-a115-bee9024c128c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c660d7cd-b579-4885-a115-bee9024c128c-istiod-ca-cert\") pod \"c660d7cd-b579-4885-a115-bee9024c128c\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") "
Oct 31 20:07:03.783093 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783072    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-data\") pod \"c660d7cd-b579-4885-a115-bee9024c128c\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") "
Oct 31 20:07:03.783153 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783092    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-credential-socket" pod "c660d7cd-b579-4885-a115-bee9024c128c" (UID: "c660d7cd-b579-4885-a115-bee9024c128c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:03.783211 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783108    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c660d7cd-b579-4885-a115-bee9024c128c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c660d7cd-b579-4885-a115-bee9024c128c-istio-podinfo\") pod \"c660d7cd-b579-4885-a115-bee9024c128c\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") "
Oct 31 20:07:03.783211 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783184    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-certs" pod "c660d7cd-b579-4885-a115-bee9024c128c" (UID: "c660d7cd-b579-4885-a115-bee9024c128c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:03.783322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783285    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-istio-token\" (UniqueName: \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-istio-token\") pod \"c660d7cd-b579-4885-a115-bee9024c128c\" (UID: \"c660d7cd-b579-4885-a115-bee9024c128c\") "
Oct 31 20:07:03.783474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783448    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-data" pod "c660d7cd-b579-4885-a115-bee9024c128c" (UID: "c660d7cd-b579-4885-a115-bee9024c128c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:03.783552 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783527    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c660d7cd-b579-4885-a115-bee9024c128c-istiod-ca-cert" pod "c660d7cd-b579-4885-a115-bee9024c128c" (UID: "c660d7cd-b579-4885-a115-bee9024c128c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:03.783894 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783868    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:03.783990 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783899    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:03.783990 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783915    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c660d7cd-b579-4885-a115-bee9024c128c-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:03.783990 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783929    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:03.784231 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.783993    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-socket" pod "c660d7cd-b579-4885-a115-bee9024c128c" (UID: "c660d7cd-b579-4885-a115-bee9024c128c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:03.785212 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.785189    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-kube-api-access-x29t8" pod "c660d7cd-b579-4885-a115-bee9024c128c" (UID: "c660d7cd-b579-4885-a115-bee9024c128c"). InnerVolumeSpecName "kube-api-access-x29t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:03.785314 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.785237    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-envoy" pod "c660d7cd-b579-4885-a115-bee9024c128c" (UID: "c660d7cd-b579-4885-a115-bee9024c128c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:03.785517 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.785487    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c660d7cd-b579-4885-a115-bee9024c128c-istio-podinfo" pod "c660d7cd-b579-4885-a115-bee9024c128c" (UID: "c660d7cd-b579-4885-a115-bee9024c128c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:07:03.785838 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.785818    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-istio-token" pod "c660d7cd-b579-4885-a115-bee9024c128c" (UID: "c660d7cd-b579-4885-a115-bee9024c128c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:03.885283 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.885242    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:03.885283 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.885275    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c660d7cd-b579-4885-a115-bee9024c128c-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:03.885283 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.885290    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c660d7cd-b579-4885-a115-bee9024c128c-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:03.885558 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.885299    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:03.885558 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:03.885310    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x29t8\" (UniqueName: \"kubernetes.io/projected/c660d7cd-b579-4885-a115-bee9024c128c-kube-api-access-x29t8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:04.041950 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.041849    5006 generic.go:356] "Generic (PLEG): container finished" podID="c660d7cd-b579-4885-a115-bee9024c128c" containerID="cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6" exitCode=0
Oct 31 20:07:04.041950 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.041900    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"
Oct 31 20:07:04.041950 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.041933    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" event={"ID":"c660d7cd-b579-4885-a115-bee9024c128c","Type":"ContainerDied","Data":"cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6"}
Oct 31 20:07:04.042597 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.041994    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz" event={"ID":"c660d7cd-b579-4885-a115-bee9024c128c","Type":"ContainerDied","Data":"4931e41b360bf29e30a4ac9ae88f9ecc08b4eaa50932d03dbfd979c7aba31d90"}
Oct 31 20:07:04.042597 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.042024    5006 scope.go:122] "RemoveContainer" containerID="cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6"
Oct 31 20:07:04.054175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.054147    5006 scope.go:122] "RemoveContainer" containerID="cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6"
Oct 31 20:07:04.054579 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:07:04.054554    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6\": container with ID starting with cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6 not found: ID does not exist" containerID="cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6"
Oct 31 20:07:04.054676 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.054587    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6"} err="failed to get container status \"cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6\": rpc error: code = NotFound desc = could not find container \"cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6\": container with ID starting with cd71f2b7967b6baba5596fb5d9d72a93b41bc6f269d535e76c0d09ebc5d926d6 not found: ID does not exist"
Oct 31 20:07:04.190479 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.190443    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:04.212284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.212249    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"]
Oct 31 20:07:04.226484 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.226457    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:04.261107 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.261075    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-rfjgz"]
Oct 31 20:07:04.290321 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.290286    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-openshift-ca\") pod \"b69ba68a-797e-4014-aa92-bd60d877492d\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") "
Oct 31 20:07:04.290527 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.290373    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cf565b-5277-48cb-bc3e-aeb9246df687-kube-api-access-6rsnx\" (UniqueName: \"kubernetes.io/projected/72cf565b-5277-48cb-bc3e-aeb9246df687-kube-api-access-6rsnx\") pod \"72cf565b-5277-48cb-bc3e-aeb9246df687\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") "
Oct 31 20:07:04.290527 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.290476    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-xks-ca\") pod \"b69ba68a-797e-4014-aa92-bd60d877492d\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") "
Oct 31 20:07:04.290713 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.290551    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-openshift-ca\") pod \"72cf565b-5277-48cb-bc3e-aeb9246df687\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") "
Oct 31 20:07:04.290713 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.290622    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69ba68a-797e-4014-aa92-bd60d877492d-kube-api-access-c9gnt\" (UniqueName: \"kubernetes.io/projected/b69ba68a-797e-4014-aa92-bd60d877492d-kube-api-access-c9gnt\") pod \"b69ba68a-797e-4014-aa92-bd60d877492d\" (UID: \"b69ba68a-797e-4014-aa92-bd60d877492d\") "
Oct 31 20:07:04.290878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.290717    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-xks-ca\") pod \"72cf565b-5277-48cb-bc3e-aeb9246df687\" (UID: \"72cf565b-5277-48cb-bc3e-aeb9246df687\") "
Oct 31 20:07:04.290878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.290795    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-openshift-ca" pod "b69ba68a-797e-4014-aa92-bd60d877492d" (UID: "b69ba68a-797e-4014-aa92-bd60d877492d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:04.290992 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.290945    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-openshift-ca" pod "72cf565b-5277-48cb-bc3e-aeb9246df687" (UID: "72cf565b-5277-48cb-bc3e-aeb9246df687"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:04.291900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.291871    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:04.292050 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.291903    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:04.292914 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.292880    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-xks-ca" pod "72cf565b-5277-48cb-bc3e-aeb9246df687" (UID: "72cf565b-5277-48cb-bc3e-aeb9246df687"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:04.293028 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.292962    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-xks-ca" pod "b69ba68a-797e-4014-aa92-bd60d877492d" (UID: "b69ba68a-797e-4014-aa92-bd60d877492d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:04.293028 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.292991    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cf565b-5277-48cb-bc3e-aeb9246df687-kube-api-access-6rsnx" pod "72cf565b-5277-48cb-bc3e-aeb9246df687" (UID: "72cf565b-5277-48cb-bc3e-aeb9246df687"). InnerVolumeSpecName "kube-api-access-6rsnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:04.293108 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.293056    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69ba68a-797e-4014-aa92-bd60d877492d-kube-api-access-c9gnt" pod "b69ba68a-797e-4014-aa92-bd60d877492d" (UID: "b69ba68a-797e-4014-aa92-bd60d877492d"). InnerVolumeSpecName "kube-api-access-c9gnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:04.393058 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.393014    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rsnx\" (UniqueName: \"kubernetes.io/projected/72cf565b-5277-48cb-bc3e-aeb9246df687-kube-api-access-6rsnx\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:04.393058 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.393054    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69ba68a-797e-4014-aa92-bd60d877492d-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:04.393280 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.393068    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9gnt\" (UniqueName: \"kubernetes.io/projected/b69ba68a-797e-4014-aa92-bd60d877492d-kube-api-access-c9gnt\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:04.393280 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:04.393083    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72cf565b-5277-48cb-bc3e-aeb9246df687-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:05.052637 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:05.052601    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"
Oct 31 20:07:05.053094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:05.052611    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w" event={"ID":"72cf565b-5277-48cb-bc3e-aeb9246df687","Type":"ContainerDied","Data":"b1cecd5df8daada95d8bae8b8efc5c399c5f5fbeab56def26e3d48fc2ba615a3"}
Oct 31 20:07:05.053094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:05.052717    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1cecd5df8daada95d8bae8b8efc5c399c5f5fbeab56def26e3d48fc2ba615a3"
Oct 31 20:07:05.054010 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:05.053989    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh" event={"ID":"b69ba68a-797e-4014-aa92-bd60d877492d","Type":"ContainerDied","Data":"0d8cadf8264abe30f4aa4cddd3aa2613d6e3a2b42f259aa019e14d9663dd8a84"}
Oct 31 20:07:05.054125 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:05.054010    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"
Oct 31 20:07:05.054125 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:05.054022    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8cadf8264abe30f4aa4cddd3aa2613d6e3a2b42f259aa019e14d9663dd8a84"
Oct 31 20:07:05.400829 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:07:05.400789    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:07:05.599197 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:05.599160    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c660d7cd-b579-4885-a115-bee9024c128c" path="/var/lib/kubelet/pods/c660d7cd-b579-4885-a115-bee9024c128c/volumes"
Oct 31 20:07:06.733692 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:06.727078    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:08.090395 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:08.090357    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:11.788090 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:11.788054    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:15.268753 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:15.268665    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:15.532738 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:15.532611    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:16.018704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:16.018630    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:16.873439 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:16.873404    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:21.683368 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:21.683320    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:21.912700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:21.912666    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:24.383686 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:24.383639    5006 scope.go:122] "RemoveContainer" containerID="17af874350d04a2f4676b04ba8d83d909a81f744ebaab7bc52dcedc0cd2482cd"
Oct 31 20:07:24.394633 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:24.394607    5006 scope.go:122] "RemoveContainer" containerID="7bc6b2c24c3887fcf84b253e275e3ef9c6774e9aceadd220676671f950db148d"
Oct 31 20:07:24.407591 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:24.407568    5006 scope.go:122] "RemoveContainer" containerID="d10bbde5aa19d6bde0a332407c605a403a895f15a573a4ac7a0698eadedb3cec"
Oct 31 20:07:26.794903 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:26.794870    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:27.216873 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:27.216807    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:28.947420 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:28.947379    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:32.119134 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:32.119097    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:35.393866 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:07:35.393826    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:07:36.961817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:36.961773    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:37.628695 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:37.628635    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:07:39.048308 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:39.048273    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"]
Oct 31 20:07:39.099626 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:39.099582    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f1d660-56mpl"]
Oct 31 20:07:39.598508 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:39.598474    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f9b171-42c4-4f58-a37b-9d2021007cd3" path="/var/lib/kubelet/pods/35f9b171-42c4-4f58-a37b-9d2021007cd3/volumes"
Oct 31 20:07:40.053753 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:40.053676    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"]
Oct 31 20:07:40.104191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:40.104154    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b32e68-zsc2k"]
Oct 31 20:07:41.598470 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:41.598436    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251545ff-dc9a-4999-9745-c4fec6d46289" path="/var/lib/kubelet/pods/251545ff-dc9a-4999-9745-c4fec6d46289/volumes"
Oct 31 20:07:41.921029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:41.920933    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:42.149526 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:42.149489    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:42.863523 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:42.863485    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"]
Oct 31 20:07:42.863951 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:42.863708    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy" containerID="cri-o://97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3" gracePeriod=30
Oct 31 20:07:44.829547 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:44.829506    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:45.053542 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:45.053499    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:46.700803 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:46.700768    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:48.143318 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.143250    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 20:07:48.156948 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.156915    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:48.268344 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268304    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-envoy\") pod \"92f49887-fe0c-440a-bdee-a73a16e56385\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") "
Oct 31 20:07:48.268553 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268377    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f49887-fe0c-440a-bdee-a73a16e56385-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f49887-fe0c-440a-bdee-a73a16e56385-istiod-ca-cert\") pod \"92f49887-fe0c-440a-bdee-a73a16e56385\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") "
Oct 31 20:07:48.268553 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268408    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-data\") pod \"92f49887-fe0c-440a-bdee-a73a16e56385\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") "
Oct 31 20:07:48.268553 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268454    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-certs\") pod \"92f49887-fe0c-440a-bdee-a73a16e56385\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") "
Oct 31 20:07:48.268553 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268507    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-kube-api-access-84prx\" (UniqueName: \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-kube-api-access-84prx\") pod \"92f49887-fe0c-440a-bdee-a73a16e56385\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") "
Oct 31 20:07:48.268785 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268594    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-credential-socket\") pod \"92f49887-fe0c-440a-bdee-a73a16e56385\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") "
Oct 31 20:07:48.268785 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268626    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-istio-token\" (UniqueName: \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-istio-token\") pod \"92f49887-fe0c-440a-bdee-a73a16e56385\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") "
Oct 31 20:07:48.268894 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268772    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-data" pod "92f49887-fe0c-440a-bdee-a73a16e56385" (UID: "92f49887-fe0c-440a-bdee-a73a16e56385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:48.268894 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268798    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92f49887-fe0c-440a-bdee-a73a16e56385-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f49887-fe0c-440a-bdee-a73a16e56385-istio-podinfo\") pod \"92f49887-fe0c-440a-bdee-a73a16e56385\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") "
Oct 31 20:07:48.268894 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268829    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-certs" pod "92f49887-fe0c-440a-bdee-a73a16e56385" (UID: "92f49887-fe0c-440a-bdee-a73a16e56385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:48.268894 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268842    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-socket\") pod \"92f49887-fe0c-440a-bdee-a73a16e56385\" (UID: \"92f49887-fe0c-440a-bdee-a73a16e56385\") "
Oct 31 20:07:48.268894 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268878    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f49887-fe0c-440a-bdee-a73a16e56385-istiod-ca-cert" pod "92f49887-fe0c-440a-bdee-a73a16e56385" (UID: "92f49887-fe0c-440a-bdee-a73a16e56385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:48.269134 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.268874    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-credential-socket" pod "92f49887-fe0c-440a-bdee-a73a16e56385" (UID: "92f49887-fe0c-440a-bdee-a73a16e56385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:48.269187 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.269157    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-socket" pod "92f49887-fe0c-440a-bdee-a73a16e56385" (UID: "92f49887-fe0c-440a-bdee-a73a16e56385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:48.269342 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.269320    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:48.269442 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.269348    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f49887-fe0c-440a-bdee-a73a16e56385-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:48.269442 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.269362    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:48.269442 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.269377    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:48.269442 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.269389    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:48.271319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.271289    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92f49887-fe0c-440a-bdee-a73a16e56385-istio-podinfo" pod "92f49887-fe0c-440a-bdee-a73a16e56385" (UID: "92f49887-fe0c-440a-bdee-a73a16e56385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:07:48.271319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.271292    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-envoy" pod "92f49887-fe0c-440a-bdee-a73a16e56385" (UID: "92f49887-fe0c-440a-bdee-a73a16e56385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:48.271462 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.271319    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-istio-token" pod "92f49887-fe0c-440a-bdee-a73a16e56385" (UID: "92f49887-fe0c-440a-bdee-a73a16e56385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:48.271462 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.271300    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-kube-api-access-84prx" pod "92f49887-fe0c-440a-bdee-a73a16e56385" (UID: "92f49887-fe0c-440a-bdee-a73a16e56385"). InnerVolumeSpecName "kube-api-access-84prx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:48.370529 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.370490    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f49887-fe0c-440a-bdee-a73a16e56385-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:48.370529 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.370520    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f49887-fe0c-440a-bdee-a73a16e56385-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:48.370529 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.370531    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84prx\" (UniqueName: \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-kube-api-access-84prx\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:48.370815 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.370541    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f49887-fe0c-440a-bdee-a73a16e56385-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:48.387162 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.387125    5006 generic.go:356] "Generic (PLEG): container finished" podID="92f49887-fe0c-440a-bdee-a73a16e56385" containerID="97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3" exitCode=0
Oct 31 20:07:48.387327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.387173    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"
Oct 31 20:07:48.387327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.387206    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" event={"ID":"92f49887-fe0c-440a-bdee-a73a16e56385","Type":"ContainerDied","Data":"97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3"}
Oct 31 20:07:48.387327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.387263    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59" event={"ID":"92f49887-fe0c-440a-bdee-a73a16e56385","Type":"ContainerDied","Data":"5f2edd4b3459c2537e7a9acca3b95975e73b75837a4d7f031dd722b216f6f2a9"}
Oct 31 20:07:48.387327 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.387285    5006 scope.go:122] "RemoveContainer" containerID="97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3"
Oct 31 20:07:48.398318 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.398295    5006 scope.go:122] "RemoveContainer" containerID="97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3"
Oct 31 20:07:48.398642 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:07:48.398620    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3\": container with ID starting with 97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3 not found: ID does not exist" containerID="97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3"
Oct 31 20:07:48.398850 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.398668    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3"} err="failed to get container status \"97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3\": rpc error: code = NotFound desc = could not find container \"97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3\": container with ID starting with 97e16db31b30cc53025f649d05906a5351485a54aabce1ac477b620e073ab5d3 not found: ID does not exist"
Oct 31 20:07:48.556734 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.556689    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"]
Oct 31 20:07:48.559555 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:48.559529    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-67h59"]
Oct 31 20:07:49.598308 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:49.598274    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" path="/var/lib/kubelet/pods/92f49887-fe0c-440a-bdee-a73a16e56385/volumes"
Oct 31 20:07:51.615155 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:51.615117    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:52.066233 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066150    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"]
Oct 31 20:07:52.066700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066686    5006 cpu_manager.go:414] "containerMap: removing container" podUID="b69ba68a-797e-4014-aa92-bd60d877492d" containerName="revoke-keys"
Oct 31 20:07:52.066759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066700    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69ba68a-797e-4014-aa92-bd60d877492d" containerName="revoke-keys"
Oct 31 20:07:52.066759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066738    5006 cpu_manager.go:414] "containerMap: removing container" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy"
Oct 31 20:07:52.066759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066744    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy"
Oct 31 20:07:52.066759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066752    5006 cpu_manager.go:414] "containerMap: removing container" podUID="72cf565b-5277-48cb-bc3e-aeb9246df687" containerName="revoke-keys"
Oct 31 20:07:52.066759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066756    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cf565b-5277-48cb-bc3e-aeb9246df687" containerName="revoke-keys"
Oct 31 20:07:52.066918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066773    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy"
Oct 31 20:07:52.066918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066778    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy"
Oct 31 20:07:52.066918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066860    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f49887-fe0c-440a-bdee-a73a16e56385" containerName="istio-proxy"
Oct 31 20:07:52.066918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066867    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cf565b-5277-48cb-bc3e-aeb9246df687" containerName="revoke-keys"
Oct 31 20:07:52.066918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066877    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c660d7cd-b579-4885-a115-bee9024c128c" containerName="istio-proxy"
Oct 31 20:07:52.066918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.066885    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69ba68a-797e-4014-aa92-bd60d877492d" containerName="revoke-keys"
Oct 31 20:07:52.070174 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.070157    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.110433 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.110397    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b208fcd6-gnx9h\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.110606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.110526    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b208fcd6-gnx9h\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.110606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.110567    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8p5\" (UniqueName: \"kubernetes.io/projected/1b369441-a0e9-4337-b7ac-cde5ba300eec-kube-api-access-zx8p5\") pod \"maas-api-revoke-sub-b208fcd6-gnx9h\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.124073 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.124044    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"]
Oct 31 20:07:52.171179 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.171143    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:07:52.171352 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.171147    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:07:52.173184 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.173169    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:07:52.173461 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.173446    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:07:52.212000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.211965    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b208fcd6-gnx9h\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.212203 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.212060    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b208fcd6-gnx9h\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.212203 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.212088    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8p5\" (UniqueName: \"kubernetes.io/projected/1b369441-a0e9-4337-b7ac-cde5ba300eec-kube-api-access-zx8p5\") pod \"maas-api-revoke-sub-b208fcd6-gnx9h\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.212772 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.212750    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b208fcd6-gnx9h\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.214503 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.214480    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b208fcd6-gnx9h\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.273297 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.273262    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8p5\" (UniqueName: \"kubernetes.io/projected/1b369441-a0e9-4337-b7ac-cde5ba300eec-kube-api-access-zx8p5\") pod \"maas-api-revoke-sub-b208fcd6-gnx9h\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.384223 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.384188    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:52.756577 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:07:52.756537    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b369441_a0e9_4337_b7ac_cde5ba300eec.slice/crio-542322faac0c2db285efa07b7e6d35b0c6e27d8040990762b53ce8e033f52922 WatchSource:0}: Error finding container 542322faac0c2db285efa07b7e6d35b0c6e27d8040990762b53ce8e033f52922: Status 404 returned error can't find the container with id 542322faac0c2db285efa07b7e6d35b0c6e27d8040990762b53ce8e033f52922
Oct 31 20:07:52.762085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:52.762045    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"]
Oct 31 20:07:53.431036 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:53.430999    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h" event={"ID":"1b369441-a0e9-4337-b7ac-cde5ba300eec","Type":"ContainerStarted","Data":"7c867c30a153733deea84e14817f8167c4b55766eb5c238a2e57cf5bc83ed21f"}
Oct 31 20:07:53.431263 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:53.431054    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h" event={"ID":"1b369441-a0e9-4337-b7ac-cde5ba300eec","Type":"ContainerStarted","Data":"542322faac0c2db285efa07b7e6d35b0c6e27d8040990762b53ce8e033f52922"}
Oct 31 20:07:53.551570 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:53.551513    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h" podStartSLOduration=1.551498043 podStartE2EDuration="1.551498043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:07:53.550360491 +0000 UTC m=+3092.079894544" watchObservedRunningTime="2026-10-31 20:07:53.551498043 +0000 UTC m=+3092.081032096"
Oct 31 20:07:54.439276 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:54.439240    5006 generic.go:356] "Generic (PLEG): container finished" podID="1b369441-a0e9-4337-b7ac-cde5ba300eec" containerID="7c867c30a153733deea84e14817f8167c4b55766eb5c238a2e57cf5bc83ed21f" exitCode=0
Oct 31 20:07:54.439753 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:54.439319    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h" event={"ID":"1b369441-a0e9-4337-b7ac-cde5ba300eec","Type":"ContainerDied","Data":"7c867c30a153733deea84e14817f8167c4b55766eb5c238a2e57cf5bc83ed21f"}
Oct 31 20:07:55.600439 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:55.600412    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:55.658222 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:55.658186    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-xks-ca\") pod \"1b369441-a0e9-4337-b7ac-cde5ba300eec\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") "
Oct 31 20:07:55.658222 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:55.658228    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-openshift-ca\") pod \"1b369441-a0e9-4337-b7ac-cde5ba300eec\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") "
Oct 31 20:07:55.658501 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:55.658274    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b369441-a0e9-4337-b7ac-cde5ba300eec-kube-api-access-zx8p5\" (UniqueName: \"kubernetes.io/projected/1b369441-a0e9-4337-b7ac-cde5ba300eec-kube-api-access-zx8p5\") pod \"1b369441-a0e9-4337-b7ac-cde5ba300eec\" (UID: \"1b369441-a0e9-4337-b7ac-cde5ba300eec\") "
Oct 31 20:07:55.658621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:55.658591    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-openshift-ca" pod "1b369441-a0e9-4337-b7ac-cde5ba300eec" (UID: "1b369441-a0e9-4337-b7ac-cde5ba300eec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:55.660461 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:55.660427    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b369441-a0e9-4337-b7ac-cde5ba300eec-kube-api-access-zx8p5" pod "1b369441-a0e9-4337-b7ac-cde5ba300eec" (UID: "1b369441-a0e9-4337-b7ac-cde5ba300eec"). InnerVolumeSpecName "kube-api-access-zx8p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:55.660593 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:55.660495    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-xks-ca" pod "1b369441-a0e9-4337-b7ac-cde5ba300eec" (UID: "1b369441-a0e9-4337-b7ac-cde5ba300eec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:55.759941 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:55.759857    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:55.759941 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:55.759889    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b369441-a0e9-4337-b7ac-cde5ba300eec-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:55.759941 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:55.759902    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8p5\" (UniqueName: \"kubernetes.io/projected/1b369441-a0e9-4337-b7ac-cde5ba300eec-kube-api-access-zx8p5\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:56.017000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:56.016903    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:56.455202 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:56.455170    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"
Oct 31 20:07:56.455388 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:56.455175    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h" event={"ID":"1b369441-a0e9-4337-b7ac-cde5ba300eec","Type":"ContainerDied","Data":"542322faac0c2db285efa07b7e6d35b0c6e27d8040990762b53ce8e033f52922"}
Oct 31 20:07:56.455388 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:56.455314    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="542322faac0c2db285efa07b7e6d35b0c6e27d8040990762b53ce8e033f52922"
Oct 31 20:07:57.472439 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:57.472401    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:57.706829 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:57.706798    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:57.939557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:57.939510    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"]
Oct 31 20:07:57.989968 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:57.989929    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a4438-pg4x8"]
Oct 31 20:07:58.074316 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:58.074285    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:58.421518 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:58.421483    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:59.598511 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:07:59.598475    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feac09ea-713a-4e42-ba72-7ac3ff003cf0" path="/var/lib/kubelet/pods/feac09ea-713a-4e42-ba72-7ac3ff003cf0/volumes"
Oct 31 20:08:02.803998 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:02.803917    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:03.157317 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.157281    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:03.254272 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.254238    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"]
Oct 31 20:08:03.254831 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.254816    5006 cpu_manager.go:414] "containerMap: removing container" podUID="1b369441-a0e9-4337-b7ac-cde5ba300eec" containerName="revoke-keys"
Oct 31 20:08:03.254901 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.254833    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b369441-a0e9-4337-b7ac-cde5ba300eec" containerName="revoke-keys"
Oct 31 20:08:03.254945 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.254928    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b369441-a0e9-4337-b7ac-cde5ba300eec" containerName="revoke-keys"
Oct 31 20:08:03.258796 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.258777    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.311010 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.310972    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"]
Oct 31 20:08:03.352015 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.351976    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.352237 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.352032    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rt2\" (UniqueName: \"kubernetes.io/projected/3849823f-6e58-43d4-944c-ba9d25160061-kube-api-access-59rt2\") pod \"maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.352237 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.352211    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.358252 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.358211    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:08:03.358252 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.358241    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:08:03.358252 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.358226    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:08:03.361899 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.361862    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:08:03.453626 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.453515    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.453837 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.453699    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.453837 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.453735    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rt2\" (UniqueName: \"kubernetes.io/projected/3849823f-6e58-43d4-944c-ba9d25160061-kube-api-access-59rt2\") pod \"maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.454388 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.454360    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.456306 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.456286    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.511988 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.511958    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rt2\" (UniqueName: \"kubernetes.io/projected/3849823f-6e58-43d4-944c-ba9d25160061-kube-api-access-59rt2\") pod \"maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.573195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.573157    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:03.953611 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:08:03.953573    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3849823f_6e58_43d4_944c_ba9d25160061.slice/crio-f5bb2881f56ab74dfad5ded1a900c2cde33f75682fd25c4f8968f0bffd0b86a3 WatchSource:0}: Error finding container f5bb2881f56ab74dfad5ded1a900c2cde33f75682fd25c4f8968f0bffd0b86a3: Status 404 returned error can't find the container with id f5bb2881f56ab74dfad5ded1a900c2cde33f75682fd25c4f8968f0bffd0b86a3
Oct 31 20:08:03.954010 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:03.953903    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"]
Oct 31 20:08:04.230205 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:04.230167    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:08:04.511566 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:04.511480    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc" event={"ID":"3849823f-6e58-43d4-944c-ba9d25160061","Type":"ContainerStarted","Data":"c62e2fb0dc3d980cc527a30523317a94ab4a295feba27eef0e692a95b8c7c0cc"}
Oct 31 20:08:04.511566 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:04.511532    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc" event={"ID":"3849823f-6e58-43d4-944c-ba9d25160061","Type":"ContainerStarted","Data":"f5bb2881f56ab74dfad5ded1a900c2cde33f75682fd25c4f8968f0bffd0b86a3"}
Oct 31 20:08:04.631427 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:04.631364    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc" podStartSLOduration=1.631349008 podStartE2EDuration="1.631349008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:08:04.631229763 +0000 UTC m=+3103.160763818" watchObservedRunningTime="2026-10-31 20:08:04.631349008 +0000 UTC m=+3103.160883062"
Oct 31 20:08:04.721407 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:04.721295    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:05.012546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:05.012507    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:05.015515 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:05.015484    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"]
Oct 31 20:08:05.015797 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:05.015763    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy" containerID="cri-o://2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b" gracePeriod=30
Oct 31 20:08:05.292356 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:05.292323    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:05.395271 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:08:05.395234    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:08:05.520116 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:05.520078    5006 generic.go:356] "Generic (PLEG): container finished" podID="3849823f-6e58-43d4-944c-ba9d25160061" containerID="c62e2fb0dc3d980cc527a30523317a94ab4a295feba27eef0e692a95b8c7c0cc" exitCode=0
Oct 31 20:08:05.520334 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:05.520171    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc" event={"ID":"3849823f-6e58-43d4-944c-ba9d25160061","Type":"ContainerDied","Data":"c62e2fb0dc3d980cc527a30523317a94ab4a295feba27eef0e692a95b8c7c0cc"}
Oct 31 20:08:06.669843 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:06.669811    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:06.699756 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:06.699722    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3849823f-6e58-43d4-944c-ba9d25160061-kube-api-access-59rt2\" (UniqueName: \"kubernetes.io/projected/3849823f-6e58-43d4-944c-ba9d25160061-kube-api-access-59rt2\") pod \"3849823f-6e58-43d4-944c-ba9d25160061\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") "
Oct 31 20:08:06.699962 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:06.699819    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-openshift-ca\") pod \"3849823f-6e58-43d4-944c-ba9d25160061\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") "
Oct 31 20:08:06.699962 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:06.699886    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-xks-ca\") pod \"3849823f-6e58-43d4-944c-ba9d25160061\" (UID: \"3849823f-6e58-43d4-944c-ba9d25160061\") "
Oct 31 20:08:06.700191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:06.700164    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-openshift-ca" pod "3849823f-6e58-43d4-944c-ba9d25160061" (UID: "3849823f-6e58-43d4-944c-ba9d25160061"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:06.702069 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:06.702038    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-xks-ca" pod "3849823f-6e58-43d4-944c-ba9d25160061" (UID: "3849823f-6e58-43d4-944c-ba9d25160061"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:06.702167 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:06.702070    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3849823f-6e58-43d4-944c-ba9d25160061-kube-api-access-59rt2" pod "3849823f-6e58-43d4-944c-ba9d25160061" (UID: "3849823f-6e58-43d4-944c-ba9d25160061"). InnerVolumeSpecName "kube-api-access-59rt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:06.801163 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:06.801069    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.801163 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:06.801098    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rt2\" (UniqueName: \"kubernetes.io/projected/3849823f-6e58-43d4-944c-ba9d25160061-kube-api-access-59rt2\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.801163 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:06.801109    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3849823f-6e58-43d4-944c-ba9d25160061-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:07.535921 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:07.535893    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"
Oct 31 20:08:07.536121 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:07.535927    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc" event={"ID":"3849823f-6e58-43d4-944c-ba9d25160061","Type":"ContainerDied","Data":"f5bb2881f56ab74dfad5ded1a900c2cde33f75682fd25c4f8968f0bffd0b86a3"}
Oct 31 20:08:07.536121 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:07.535989    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5bb2881f56ab74dfad5ded1a900c2cde33f75682fd25c4f8968f0bffd0b86a3"
Oct 31 20:08:07.692145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:07.692095    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"]
Oct 31 20:08:07.733015 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:07.732977    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"]
Oct 31 20:08:07.942508 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:07.942474    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:08.533014 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:08.532981    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:09.492509 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:09.492459    5006 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:08:09.492956 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:09.492557    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:08:10.304161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.304131    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 20:08:10.335039 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335006    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-socket\") pod \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") "
Oct 31 20:08:10.335039 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335042    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-podinfo\") pod \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") "
Oct 31 20:08:10.335259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335072    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-data\") pod \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") "
Oct 31 20:08:10.335259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335113    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-kube-api-access-z66g9\" (UniqueName: \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-kube-api-access-z66g9\") pod \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") "
Oct 31 20:08:10.335259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335144    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-envoy\") pod \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") "
Oct 31 20:08:10.335259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335166    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istiod-ca-cert\") pod \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") "
Oct 31 20:08:10.335259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335236    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-token\" (UniqueName: \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-token\") pod \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") "
Oct 31 20:08:10.335532 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335287    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-credential-socket\") pod \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") "
Oct 31 20:08:10.335532 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335324    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-certs\") pod \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\" (UID: \"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125\") "
Oct 31 20:08:10.335532 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335327    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-socket" pod "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" (UID: "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:10.335766 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335589    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-credential-socket" pod "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" (UID: "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:10.335766 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335731    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istiod-ca-cert" pod "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" (UID: "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:10.335878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335845    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-certs" pod "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" (UID: "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:10.335944 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.335919    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-data" pod "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" (UID: "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:10.336104 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.336085    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:10.336174 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.336108    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:10.336174 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.336124    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:10.336174 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.336138    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:10.336174 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.336150    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:10.337707 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.337681    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-kube-api-access-z66g9" pod "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" (UID: "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125"). InnerVolumeSpecName "kube-api-access-z66g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:10.337903 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.337884    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-podinfo" pod "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" (UID: "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:08:10.337975 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.337923    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-token" pod "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" (UID: "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:10.338034 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.337968    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-envoy" pod "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" (UID: "bfd4ccb3-3226-42c4-a0d9-a6d7ac564125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:10.437136 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.437092    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:10.437136 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.437128    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:10.437136 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.437138    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:10.437136 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.437147    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66g9\" (UniqueName: \"kubernetes.io/projected/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125-kube-api-access-z66g9\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:10.558971 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.558875    5006 generic.go:356] "Generic (PLEG): container finished" podID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerID="2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b" exitCode=0
Oct 31 20:08:10.558971 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.558927    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"
Oct 31 20:08:10.558971 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.558941    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" event={"ID":"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125","Type":"ContainerDied","Data":"2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b"}
Oct 31 20:08:10.559567 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.559008    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg" event={"ID":"bfd4ccb3-3226-42c4-a0d9-a6d7ac564125","Type":"ContainerDied","Data":"df758f99e28c5422defbddbbd2838c6f9debda5aa365c7ae291ac3f52c10e104"}
Oct 31 20:08:10.559567 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.559038    5006 scope.go:122] "RemoveContainer" containerID="2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b"
Oct 31 20:08:10.570507 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.570486    5006 scope.go:122] "RemoveContainer" containerID="2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b"
Oct 31 20:08:10.570854 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:08:10.570833    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b\": container with ID starting with 2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b not found: ID does not exist" containerID="2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b"
Oct 31 20:08:10.570930 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.570865    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b"} err="failed to get container status \"2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b\": rpc error: code = NotFound desc = could not find container \"2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b\": container with ID starting with 2b25a7adc6b69d1f10e9dc4a362a695174984756f972fe1c8e1c02c095398f4b not found: ID does not exist"
Oct 31 20:08:10.726056 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.726017    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"]
Oct 31 20:08:10.735498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:10.735447    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-qc6bg"]
Oct 31 20:08:11.598756 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:11.598722    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" path="/var/lib/kubelet/pods/bfd4ccb3-3226-42c4-a0d9-a6d7ac564125/volumes"
Oct 31 20:08:13.586022 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:13.585987    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:14.828282 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:14.828243    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"]
Oct 31 20:08:14.946059 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:14.946026    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"]
Oct 31 20:08:15.052514 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:15.052480    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"]
Oct 31 20:08:15.055255 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:15.055233    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3063f26-wsk6x"]
Oct 31 20:08:15.289921 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:15.289885    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:15.508866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:15.508820    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:15.602797 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:15.602750    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32057031-e5f3-4184-924d-7ba28c0d3e5f" path="/var/lib/kubelet/pods/32057031-e5f3-4184-924d-7ba28c0d3e5f/volumes"
Oct 31 20:08:18.543392 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.543358    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:18.694187 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.694151    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"]
Oct 31 20:08:18.694770 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.694752    5006 cpu_manager.go:414] "containerMap: removing container" podUID="3849823f-6e58-43d4-944c-ba9d25160061" containerName="revoke-keys"
Oct 31 20:08:18.694770 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.694769    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3849823f-6e58-43d4-944c-ba9d25160061" containerName="revoke-keys"
Oct 31 20:08:18.694906 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.694812    5006 cpu_manager.go:414] "containerMap: removing container" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy"
Oct 31 20:08:18.694906 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.694818    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy"
Oct 31 20:08:18.694987 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.694924    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd4ccb3-3226-42c4-a0d9-a6d7ac564125" containerName="istio-proxy"
Oct 31 20:08:18.694987 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.694935    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3849823f-6e58-43d4-944c-ba9d25160061" containerName="revoke-keys"
Oct 31 20:08:18.699710 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.699686    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.732847 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.732811    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-dshm\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.733057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.732865    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23dc8445-ca59-4827-a21d-c11fe709e600-tls-certs\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.733057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.732949    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-tmp-dir\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.733057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.732982    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-home\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.733182 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.733141    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrwmg\" (UniqueName: \"kubernetes.io/projected/23dc8445-ca59-4827-a21d-c11fe709e600-kube-api-access-nrwmg\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.733290 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.733271    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-model-cache\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.755566 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.755527    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"]
Oct 31 20:08:18.802722 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.802628    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef44dd\"/\"default-dockercfg-tg2kz\""
Oct 31 20:08:18.802722 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.802645    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef44dd\"/\"kube-root-ca.crt\""
Oct 31 20:08:18.803471 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.803452    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef44dd\"/\"openshift-service-ca.crt\""
Oct 31 20:08:18.803578 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.803476    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef44dd\"/\"test-model-ef44dd-kserve-self-signed-certs\""
Oct 31 20:08:18.834134 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.834093    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-dshm\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.834546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.834154    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23dc8445-ca59-4827-a21d-c11fe709e600-tls-certs\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.834546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.834192    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-tmp-dir\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.834546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.834394    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-home\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.834546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.834477    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrwmg\" (UniqueName: \"kubernetes.io/projected/23dc8445-ca59-4827-a21d-c11fe709e600-kube-api-access-nrwmg\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.835068 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.834553    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-model-cache\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.837100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.836588    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-home\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.841616 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.841570    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-tmp-dir\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.841856 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.841780    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-dshm\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.842282 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.842252    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23dc8445-ca59-4827-a21d-c11fe709e600-tls-certs\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.842439 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.842408    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-model-cache\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:18.889760 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.889726    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:18.896177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:18.896153    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrwmg\" (UniqueName: \"kubernetes.io/projected/23dc8445-ca59-4827-a21d-c11fe709e600-kube-api-access-nrwmg\") pod \"test-model-ef44dd-kserve-bf8884dfc-vrkrz\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") " pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:19.012832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:19.012791    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:19.403570 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:08:19.403536    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23dc8445_ca59_4827_a21d_c11fe709e600.slice/crio-94283b77996914faa93fc656d34bb4c217934fe577637bdd31b43993122d9c08 WatchSource:0}: Error finding container 94283b77996914faa93fc656d34bb4c217934fe577637bdd31b43993122d9c08: Status 404 returned error can't find the container with id 94283b77996914faa93fc656d34bb4c217934fe577637bdd31b43993122d9c08
Oct 31 20:08:19.407139 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:19.407110    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"]
Oct 31 20:08:19.628321 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:19.628283    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz" event={"ID":"23dc8445-ca59-4827-a21d-c11fe709e600","Type":"ContainerStarted","Data":"8a44d92374a7f93d5702f49e73cfc77d0948418e41a149607c42dd3663a9abec"}
Oct 31 20:08:19.628763 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:19.628341    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz" event={"ID":"23dc8445-ca59-4827-a21d-c11fe709e600","Type":"ContainerStarted","Data":"94283b77996914faa93fc656d34bb4c217934fe577637bdd31b43993122d9c08"}
Oct 31 20:08:19.751911 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:19.751813    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz" podStartSLOduration=1.751796549 podStartE2EDuration="1.751796549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:08:19.749453326 +0000 UTC m=+3118.278987380" watchObservedRunningTime="2026-10-31 20:08:19.751796549 +0000 UTC m=+3118.281330655"
Oct 31 20:08:19.829334 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:19.829282    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9" podUID="33d0fa26-cffd-44a1-b12a-ba9d6207d3d7" containerName="payload-processing" containerID="cri-o://d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46" gracePeriod=25
Oct 31 20:08:19.947216 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:19.947165    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn" podUID="e1ad6ab8-45f3-4ae4-88f8-7572147cabb7" containerName="payload-pre-processing" containerID="cri-o://81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d" gracePeriod=25
Oct 31 20:08:20.013156 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:20.013073    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:22.714126 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:22.714081    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main" containerID="cri-o://cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393" gracePeriod=45
Oct 31 20:08:22.754078 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:22.754020    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main" containerID="cri-o://5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc" gracePeriod=45
Oct 31 20:08:22.984769 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:22.984619    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn" podUID="e1ad6ab8-45f3-4ae4-88f8-7572147cabb7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:23.031243 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.031215    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 20:08:23.065292 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.065249    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 20:08:23.084666 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.084618    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-home\") pod \"e9f8518c-5924-44d3-8a07-e6578830d3b6\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") "
Oct 31 20:08:23.084858 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.084745    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-dshm\") pod \"e9f8518c-5924-44d3-8a07-e6578830d3b6\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") "
Oct 31 20:08:23.084858 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.084787    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-model-cache\") pod \"e9f8518c-5924-44d3-8a07-e6578830d3b6\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") "
Oct 31 20:08:23.084956 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.084879    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-tmp-dir\") pod \"e9f8518c-5924-44d3-8a07-e6578830d3b6\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") "
Oct 31 20:08:23.084956 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.084939    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-home" pod "e9f8518c-5924-44d3-8a07-e6578830d3b6" (UID: "e9f8518c-5924-44d3-8a07-e6578830d3b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:23.085095 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.085076    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f8518c-5924-44d3-8a07-e6578830d3b6-kube-api-access-7z7z7\" (UniqueName: \"kubernetes.io/projected/e9f8518c-5924-44d3-8a07-e6578830d3b6-kube-api-access-7z7z7\") pod \"e9f8518c-5924-44d3-8a07-e6578830d3b6\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") "
Oct 31 20:08:23.085157 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.085095    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-model-cache" pod "e9f8518c-5924-44d3-8a07-e6578830d3b6" (UID: "e9f8518c-5924-44d3-8a07-e6578830d3b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:23.085157 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.085130    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f8518c-5924-44d3-8a07-e6578830d3b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8518c-5924-44d3-8a07-e6578830d3b6-tls-certs\") pod \"e9f8518c-5924-44d3-8a07-e6578830d3b6\" (UID: \"e9f8518c-5924-44d3-8a07-e6578830d3b6\") "
Oct 31 20:08:23.085346 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.085130    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-tmp-dir" pod "e9f8518c-5924-44d3-8a07-e6578830d3b6" (UID: "e9f8518c-5924-44d3-8a07-e6578830d3b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:23.085547 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.085530    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.085621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.085550    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.085621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.085560    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.087091 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.087068    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-dshm" pod "e9f8518c-5924-44d3-8a07-e6578830d3b6" (UID: "e9f8518c-5924-44d3-8a07-e6578830d3b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:23.087585 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.087560    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f8518c-5924-44d3-8a07-e6578830d3b6-tls-certs" pod "e9f8518c-5924-44d3-8a07-e6578830d3b6" (UID: "e9f8518c-5924-44d3-8a07-e6578830d3b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:23.087708 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.087598    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f8518c-5924-44d3-8a07-e6578830d3b6-kube-api-access-7z7z7" pod "e9f8518c-5924-44d3-8a07-e6578830d3b6" (UID: "e9f8518c-5924-44d3-8a07-e6578830d3b6"). InnerVolumeSpecName "kube-api-access-7z7z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:23.186407 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.186371    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-home\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-home\") pod \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") "
Oct 31 20:08:23.186613 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.186449    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tmp-dir\") pod \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") "
Oct 31 20:08:23.186613 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.186550    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-model-cache\") pod \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") "
Oct 31 20:08:23.186613 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.186607    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8f7246-2e48-4496-9934-c5eb5d4b3134-kube-api-access-w86gk\" (UniqueName: \"kubernetes.io/projected/8d8f7246-2e48-4496-9934-c5eb5d4b3134-kube-api-access-w86gk\") pod \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") "
Oct 31 20:08:23.186878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.186713    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tls-certs\") pod \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") "
Oct 31 20:08:23.186878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.186742    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-dshm\") pod \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\" (UID: \"8d8f7246-2e48-4496-9934-c5eb5d4b3134\") "
Oct 31 20:08:23.186878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.186737    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-home" pod "8d8f7246-2e48-4496-9934-c5eb5d4b3134" (UID: "8d8f7246-2e48-4496-9934-c5eb5d4b3134"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:23.186878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.186807    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tmp-dir" pod "8d8f7246-2e48-4496-9934-c5eb5d4b3134" (UID: "8d8f7246-2e48-4496-9934-c5eb5d4b3134"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:23.186878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.186868    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-model-cache" pod "8d8f7246-2e48-4496-9934-c5eb5d4b3134" (UID: "8d8f7246-2e48-4496-9934-c5eb5d4b3134"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:23.187333 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.187219    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7z7\" (UniqueName: \"kubernetes.io/projected/e9f8518c-5924-44d3-8a07-e6578830d3b6-kube-api-access-7z7z7\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.187333 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.187240    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8518c-5924-44d3-8a07-e6578830d3b6-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.187333 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.187253    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.187333 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.187265    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.187333 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.187276    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8518c-5924-44d3-8a07-e6578830d3b6-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.187333 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.187288    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.189001 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.188972    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tls-certs" pod "8d8f7246-2e48-4496-9934-c5eb5d4b3134" (UID: "8d8f7246-2e48-4496-9934-c5eb5d4b3134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:23.189403 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.189378    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8f7246-2e48-4496-9934-c5eb5d4b3134-kube-api-access-w86gk" pod "8d8f7246-2e48-4496-9934-c5eb5d4b3134" (UID: "8d8f7246-2e48-4496-9934-c5eb5d4b3134"). InnerVolumeSpecName "kube-api-access-w86gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:23.189552 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.189534    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-dshm" pod "8d8f7246-2e48-4496-9934-c5eb5d4b3134" (UID: "8d8f7246-2e48-4496-9934-c5eb5d4b3134"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:23.288342 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.288256    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w86gk\" (UniqueName: \"kubernetes.io/projected/8d8f7246-2e48-4496-9934-c5eb5d4b3134-kube-api-access-w86gk\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.288342 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.288288    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8f7246-2e48-4496-9934-c5eb5d4b3134-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.288342 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.288298    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d8f7246-2e48-4496-9934-c5eb5d4b3134-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:23.500145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.500106    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:23.658796 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.658757    5006 generic.go:356] "Generic (PLEG): container finished" podID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerID="5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc" exitCode=0
Oct 31 20:08:23.658988 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.658804    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"
Oct 31 20:08:23.658988 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.658840    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" event={"ID":"8d8f7246-2e48-4496-9934-c5eb5d4b3134","Type":"ContainerDied","Data":"5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc"}
Oct 31 20:08:23.658988 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.658878    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8" event={"ID":"8d8f7246-2e48-4496-9934-c5eb5d4b3134","Type":"ContainerDied","Data":"e8017663889458ba67e662b295fd4e448b9971714c688ce2cc043df83ba4d8fe"}
Oct 31 20:08:23.658988 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.658895    5006 scope.go:122] "RemoveContainer" containerID="5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc"
Oct 31 20:08:23.661014 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.660986    5006 generic.go:356] "Generic (PLEG): container finished" podID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerID="cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393" exitCode=0
Oct 31 20:08:23.661204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.661032    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"
Oct 31 20:08:23.661204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.661102    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" event={"ID":"e9f8518c-5924-44d3-8a07-e6578830d3b6","Type":"ContainerDied","Data":"cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393"}
Oct 31 20:08:23.661204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.661131    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x" event={"ID":"e9f8518c-5924-44d3-8a07-e6578830d3b6","Type":"ContainerDied","Data":"5f108dda5980d5d2a541e012130fb5e92301d7432db9297b54e95a7b65c576f3"}
Oct 31 20:08:23.669780 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.669759    5006 scope.go:122] "RemoveContainer" containerID="5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc"
Oct 31 20:08:23.670099 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:08:23.670074    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc\": container with ID starting with 5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc not found: ID does not exist" containerID="5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc"
Oct 31 20:08:23.670194 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.670111    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc"} err="failed to get container status \"5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc\": rpc error: code = NotFound desc = could not find container \"5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc\": container with ID starting with 5f114ca8e74f01e43acac30ce0dbff4a905388a6bb6007c4ee0a45e3575e3fdc not found: ID does not exist"
Oct 31 20:08:23.670194 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.670136    5006 scope.go:122] "RemoveContainer" containerID="cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393"
Oct 31 20:08:23.679817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.679787    5006 scope.go:122] "RemoveContainer" containerID="cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393"
Oct 31 20:08:23.680093 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:08:23.680072    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393\": container with ID starting with cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393 not found: ID does not exist" containerID="cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393"
Oct 31 20:08:23.680137 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.680102    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393"} err="failed to get container status \"cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393\": rpc error: code = NotFound desc = could not find container \"cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393\": container with ID starting with cb0331dcd1c26a177ea05e50977b7d3bb262a4c109fe5ed4a53c187ed211f393 not found: ID does not exist"
Oct 31 20:08:23.826381 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.826345    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"]
Oct 31 20:08:23.829686 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.829637    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/simulator-w3-662be1-kserve-5b6688c44d-5cxs8"]
Oct 31 20:08:23.986957 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.986829    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"]
Oct 31 20:08:23.989219 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.989182    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9" podUID="33d0fa26-cffd-44a1-b12a-ba9d6207d3d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:23.990274 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:23.990249    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-2-w3-662be1-kserve-7487cbdd7-qqd5x"]
Oct 31 20:08:24.473549 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:24.473508    5006 scope.go:122] "RemoveContainer" containerID="129d0b684529927e1af6df32a55eb979dc0044bb9728c1b13ebf6dba291e9086"
Oct 31 20:08:24.483901 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:24.483879    5006 scope.go:122] "RemoveContainer" containerID="2e16a94b06aec8de7982048691c1292cad9d14b40bcb34ccea36e3de02eb3ea1"
Oct 31 20:08:24.493623 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:24.493601    5006 scope.go:122] "RemoveContainer" containerID="8c8b6eaae6a925460f6ed5e8c02f72fb9e8e7fd25c2b19f14a209cedc299d201"
Oct 31 20:08:24.503342 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:24.503315    5006 scope.go:122] "RemoveContainer" containerID="9bce21e1b04b0400afc0991c7a2d45c60a5ec5ca9172b0b28b8a9f2a9e91a16d"
Oct 31 20:08:25.598154 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:25.598110    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" path="/var/lib/kubelet/pods/8d8f7246-2e48-4496-9934-c5eb5d4b3134/volumes"
Oct 31 20:08:25.598820 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:25.598798    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" path="/var/lib/kubelet/pods/e9f8518c-5924-44d3-8a07-e6578830d3b6/volumes"
Oct 31 20:08:26.403464 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:26.403420    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:08:28.714677 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:28.702218    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:29.013828 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:29.013719    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:29.035429 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:29.035391    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:29.265970 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:29.265869    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:29.494372 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:29.494328    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:29.721214 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:29.721183    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:08:32.920418 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:32.920374    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"]
Oct 31 20:08:32.920990 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:32.920974    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main"
Oct 31 20:08:32.921038 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:32.920992    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main"
Oct 31 20:08:32.921038 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:32.921010    5006 cpu_manager.go:414] "containerMap: removing container" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main"
Oct 31 20:08:32.921038 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:32.921015    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main"
Oct 31 20:08:32.921147 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:32.921118    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f8518c-5924-44d3-8a07-e6578830d3b6" containerName="main"
Oct 31 20:08:32.921147 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:32.921130    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8f7246-2e48-4496-9934-c5eb5d4b3134" containerName="main"
Oct 31 20:08:32.926662 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:32.926633    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:32.984536 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:32.984499    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn" podUID="e1ad6ab8-45f3-4ae4-88f8-7572147cabb7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:33.019793 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.019752    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55wt\" (UniqueName: \"kubernetes.io/projected/52a1c6f3-45cd-4d90-8429-67fa43199198-kube-api-access-w55wt\") pod \"maas-api-revoke-sub-b3c81605-lp6q7\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:33.020017 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.019921    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c81605-lp6q7\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:33.020094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.020066    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c81605-lp6q7\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:33.030531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.030489    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:08:33.030531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.030512    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:08:33.030531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.030516    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:08:33.030788 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.030712    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:08:33.033586 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.033563    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"]
Oct 31 20:08:33.036313 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.036277    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"]
Oct 31 20:08:33.040509 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.040478    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.088049 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.088011    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"]
Oct 31 20:08:33.120739 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.120701    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c81605-lp6q7\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:33.120891 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.120752    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55wt\" (UniqueName: \"kubernetes.io/projected/52a1c6f3-45cd-4d90-8429-67fa43199198-kube-api-access-w55wt\") pod \"maas-api-revoke-sub-b3c81605-lp6q7\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:33.120891 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.120826    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c81605-lp6q7\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:33.121474 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.121449    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c81605-lp6q7\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:33.123152 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.123128    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c81605-lp6q7\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:33.178265 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.178188    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55wt\" (UniqueName: \"kubernetes.io/projected/52a1c6f3-45cd-4d90-8429-67fa43199198-kube-api-access-w55wt\") pod \"maas-api-revoke-sub-b3c81605-lp6q7\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:33.222317 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.222273    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c92bd85d-l2wdj\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.222317 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.222324    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdpbf\" (UniqueName: \"kubernetes.io/projected/9327afce-2584-4b26-97a8-85cc442e6bc2-kube-api-access-jdpbf\") pod \"maas-api-revoke-sub-c92bd85d-l2wdj\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.222551 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.222417    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c92bd85d-l2wdj\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.241207 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.241170    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:33.323322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.323277    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdpbf\" (UniqueName: \"kubernetes.io/projected/9327afce-2584-4b26-97a8-85cc442e6bc2-kube-api-access-jdpbf\") pod \"maas-api-revoke-sub-c92bd85d-l2wdj\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.323499 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.323381    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c92bd85d-l2wdj\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.323570 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.323554    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c92bd85d-l2wdj\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.324402 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.324378    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c92bd85d-l2wdj\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.326442 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.326413    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c92bd85d-l2wdj\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.381208 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.381168    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdpbf\" (UniqueName: \"kubernetes.io/projected/9327afce-2584-4b26-97a8-85cc442e6bc2-kube-api-access-jdpbf\") pod \"maas-api-revoke-sub-c92bd85d-l2wdj\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.608360 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:08:33.608331    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a1c6f3_45cd_4d90_8429_67fa43199198.slice/crio-365ec53a0a48416b8413b741b60a946d458833825139351b629672d155918b93 WatchSource:0}: Error finding container 365ec53a0a48416b8413b741b60a946d458833825139351b629672d155918b93: Status 404 returned error can't find the container with id 365ec53a0a48416b8413b741b60a946d458833825139351b629672d155918b93
Oct 31 20:08:33.611885 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.611859    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"]
Oct 31 20:08:33.655578 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.655544    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:33.735251 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.735209    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7" event={"ID":"52a1c6f3-45cd-4d90-8429-67fa43199198","Type":"ContainerStarted","Data":"365ec53a0a48416b8413b741b60a946d458833825139351b629672d155918b93"}
Oct 31 20:08:33.911576 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.911539    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:33.989404 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:33.989357    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9" podUID="33d0fa26-cffd-44a1-b12a-ba9d6207d3d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:34.027146 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:08:34.027114    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9327afce_2584_4b26_97a8_85cc442e6bc2.slice/crio-73bdf24241babaa5c89ff36d6797ec2e8153237407a5d708136052362ae15003 WatchSource:0}: Error finding container 73bdf24241babaa5c89ff36d6797ec2e8153237407a5d708136052362ae15003: Status 404 returned error can't find the container with id 73bdf24241babaa5c89ff36d6797ec2e8153237407a5d708136052362ae15003
Oct 31 20:08:34.029616 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:34.029588    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"]
Oct 31 20:08:34.744315 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:34.744279    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj" event={"ID":"9327afce-2584-4b26-97a8-85cc442e6bc2","Type":"ContainerStarted","Data":"d34d9476cbb3e4a9962b9006ff897e0b4f18971d3b30d23fef065be5dc145143"}
Oct 31 20:08:34.744479 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:34.744337    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj" event={"ID":"9327afce-2584-4b26-97a8-85cc442e6bc2","Type":"ContainerStarted","Data":"73bdf24241babaa5c89ff36d6797ec2e8153237407a5d708136052362ae15003"}
Oct 31 20:08:34.745930 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:34.745901    5006 generic.go:356] "Generic (PLEG): container finished" podID="52a1c6f3-45cd-4d90-8429-67fa43199198" containerID="c4417a2d99fa8f2e04bbaa34d40629a1ee623a2f73c44e60204aa1220f3bd4a6" exitCode=0
Oct 31 20:08:34.746084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:34.745985    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7" event={"ID":"52a1c6f3-45cd-4d90-8429-67fa43199198","Type":"ContainerDied","Data":"c4417a2d99fa8f2e04bbaa34d40629a1ee623a2f73c44e60204aa1220f3bd4a6"}
Oct 31 20:08:34.858504 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:34.858437    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj" podStartSLOduration=2.858415993 podStartE2EDuration="2.858415993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:08:34.856752919 +0000 UTC m=+3133.386286972" watchObservedRunningTime="2026-10-31 20:08:34.858415993 +0000 UTC m=+3133.387950047"
Oct 31 20:08:35.394250 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:08:35.394213    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:08:35.755903 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:35.755807    5006 generic.go:356] "Generic (PLEG): container finished" podID="9327afce-2584-4b26-97a8-85cc442e6bc2" containerID="d34d9476cbb3e4a9962b9006ff897e0b4f18971d3b30d23fef065be5dc145143" exitCode=0
Oct 31 20:08:35.755903 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:35.755888    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj" event={"ID":"9327afce-2584-4b26-97a8-85cc442e6bc2","Type":"ContainerDied","Data":"d34d9476cbb3e4a9962b9006ff897e0b4f18971d3b30d23fef065be5dc145143"}
Oct 31 20:08:35.915697 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:35.915667    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:36.055723 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.055588    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a1c6f3-45cd-4d90-8429-67fa43199198-kube-api-access-w55wt\" (UniqueName: \"kubernetes.io/projected/52a1c6f3-45cd-4d90-8429-67fa43199198-kube-api-access-w55wt\") pod \"52a1c6f3-45cd-4d90-8429-67fa43199198\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") "
Oct 31 20:08:36.055723 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.055690    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-openshift-ca\") pod \"52a1c6f3-45cd-4d90-8429-67fa43199198\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") "
Oct 31 20:08:36.055983 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.055785    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-xks-ca\") pod \"52a1c6f3-45cd-4d90-8429-67fa43199198\" (UID: \"52a1c6f3-45cd-4d90-8429-67fa43199198\") "
Oct 31 20:08:36.056112 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.056087    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-openshift-ca" pod "52a1c6f3-45cd-4d90-8429-67fa43199198" (UID: "52a1c6f3-45cd-4d90-8429-67fa43199198"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:36.056291 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.056269    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:36.057926 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.057890    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-xks-ca" pod "52a1c6f3-45cd-4d90-8429-67fa43199198" (UID: "52a1c6f3-45cd-4d90-8429-67fa43199198"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:36.058038 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.057996    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a1c6f3-45cd-4d90-8429-67fa43199198-kube-api-access-w55wt" pod "52a1c6f3-45cd-4d90-8429-67fa43199198" (UID: "52a1c6f3-45cd-4d90-8429-67fa43199198"). InnerVolumeSpecName "kube-api-access-w55wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:36.157590 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.157548    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55wt\" (UniqueName: \"kubernetes.io/projected/52a1c6f3-45cd-4d90-8429-67fa43199198-kube-api-access-w55wt\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:36.157590 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.157584    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a1c6f3-45cd-4d90-8429-67fa43199198-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:36.764509 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.764475    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"
Oct 31 20:08:36.764991 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.764516    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7" event={"ID":"52a1c6f3-45cd-4d90-8429-67fa43199198","Type":"ContainerDied","Data":"365ec53a0a48416b8413b741b60a946d458833825139351b629672d155918b93"}
Oct 31 20:08:36.764991 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.764573    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="365ec53a0a48416b8413b741b60a946d458833825139351b629672d155918b93"
Oct 31 20:08:36.924189 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:36.924161    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:37.068766 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.068678    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-openshift-ca\") pod \"9327afce-2584-4b26-97a8-85cc442e6bc2\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") "
Oct 31 20:08:37.068766 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.068746    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-xks-ca\") pod \"9327afce-2584-4b26-97a8-85cc442e6bc2\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") "
Oct 31 20:08:37.069028 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.068952    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9327afce-2584-4b26-97a8-85cc442e6bc2-kube-api-access-jdpbf\" (UniqueName: \"kubernetes.io/projected/9327afce-2584-4b26-97a8-85cc442e6bc2-kube-api-access-jdpbf\") pod \"9327afce-2584-4b26-97a8-85cc442e6bc2\" (UID: \"9327afce-2584-4b26-97a8-85cc442e6bc2\") "
Oct 31 20:08:37.069168 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.069147    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-openshift-ca" pod "9327afce-2584-4b26-97a8-85cc442e6bc2" (UID: "9327afce-2584-4b26-97a8-85cc442e6bc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:37.069446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.069422    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:37.071206 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.071174    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9327afce-2584-4b26-97a8-85cc442e6bc2-kube-api-access-jdpbf" pod "9327afce-2584-4b26-97a8-85cc442e6bc2" (UID: "9327afce-2584-4b26-97a8-85cc442e6bc2"). InnerVolumeSpecName "kube-api-access-jdpbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:37.071206 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.071179    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-xks-ca" pod "9327afce-2584-4b26-97a8-85cc442e6bc2" (UID: "9327afce-2584-4b26-97a8-85cc442e6bc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:37.170911 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.170866    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9327afce-2584-4b26-97a8-85cc442e6bc2-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:37.170911 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.170901    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdpbf\" (UniqueName: \"kubernetes.io/projected/9327afce-2584-4b26-97a8-85cc442e6bc2-kube-api-access-jdpbf\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:37.773102 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.773072    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"
Oct 31 20:08:37.773621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.773079    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj" event={"ID":"9327afce-2584-4b26-97a8-85cc442e6bc2","Type":"ContainerDied","Data":"73bdf24241babaa5c89ff36d6797ec2e8153237407a5d708136052362ae15003"}
Oct 31 20:08:37.773621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:37.773199    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73bdf24241babaa5c89ff36d6797ec2e8153237407a5d708136052362ae15003"
Oct 31 20:08:38.801005 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:38.800971    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:40.117137 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.117108    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 20:08:40.196046 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.196006    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-plugins-config-volume\") pod \"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7\" (UID: \"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7\") "
Oct 31 20:08:40.196709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.196314    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-kube-api-access-t9vhl\" (UniqueName: \"kubernetes.io/projected/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-kube-api-access-t9vhl\") pod \"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7\" (UID: \"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7\") "
Oct 31 20:08:40.196709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.196538    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-plugins-config-volume" pod "33d0fa26-cffd-44a1-b12a-ba9d6207d3d7" (UID: "33d0fa26-cffd-44a1-b12a-ba9d6207d3d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:40.196877 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.196838    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:40.198631 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.198607    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-kube-api-access-t9vhl" pod "33d0fa26-cffd-44a1-b12a-ba9d6207d3d7" (UID: "33d0fa26-cffd-44a1-b12a-ba9d6207d3d7"). InnerVolumeSpecName "kube-api-access-t9vhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:40.207605 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.207581    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 20:08:40.297293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.297258    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-kube-api-access-mrcz5\" (UniqueName: \"kubernetes.io/projected/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-kube-api-access-mrcz5\") pod \"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7\" (UID: \"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7\") "
Oct 31 20:08:40.297457 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.297316    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-plugins-config-volume\") pod \"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7\" (UID: \"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7\") "
Oct 31 20:08:40.297679 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.297639    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9vhl\" (UniqueName: \"kubernetes.io/projected/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7-kube-api-access-t9vhl\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:40.297782 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.297756    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-plugins-config-volume" pod "e1ad6ab8-45f3-4ae4-88f8-7572147cabb7" (UID: "e1ad6ab8-45f3-4ae4-88f8-7572147cabb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:40.299458 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.299439    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-kube-api-access-mrcz5" pod "e1ad6ab8-45f3-4ae4-88f8-7572147cabb7" (UID: "e1ad6ab8-45f3-4ae4-88f8-7572147cabb7"). InnerVolumeSpecName "kube-api-access-mrcz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:40.398674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.398559    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrcz5\" (UniqueName: \"kubernetes.io/projected/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-kube-api-access-mrcz5\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:40.398674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.398589    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:40.794122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.794029    5006 generic.go:356] "Generic (PLEG): container finished" podID="33d0fa26-cffd-44a1-b12a-ba9d6207d3d7" containerID="d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46" exitCode=0
Oct 31 20:08:40.794122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.794087    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"
Oct 31 20:08:40.794360 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.794119    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9" event={"ID":"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7","Type":"ContainerDied","Data":"d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46"}
Oct 31 20:08:40.794360 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.794175    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9" event={"ID":"33d0fa26-cffd-44a1-b12a-ba9d6207d3d7","Type":"ContainerDied","Data":"0627cc5259e38e55392aed17ea948301f146c3bb279e6de44a39e426a662a22a"}
Oct 31 20:08:40.794360 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.794191    5006 scope.go:122] "RemoveContainer" containerID="d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46"
Oct 31 20:08:40.795582 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.795554    5006 generic.go:356] "Generic (PLEG): container finished" podID="e1ad6ab8-45f3-4ae4-88f8-7572147cabb7" containerID="81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d" exitCode=0
Oct 31 20:08:40.795746 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.795620    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"
Oct 31 20:08:40.795812 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.795768    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn" event={"ID":"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7","Type":"ContainerDied","Data":"81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d"}
Oct 31 20:08:40.795883 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.795845    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn" event={"ID":"e1ad6ab8-45f3-4ae4-88f8-7572147cabb7","Type":"ContainerDied","Data":"2dfd6606ef3b79ba9d061397582be975a7fe8e96e8c984b21baf7320f1f8f219"}
Oct 31 20:08:40.806739 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.806719    5006 scope.go:122] "RemoveContainer" containerID="d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46"
Oct 31 20:08:40.807137 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:08:40.807099    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46\": container with ID starting with d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46 not found: ID does not exist" containerID="d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46"
Oct 31 20:08:40.807247 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.807139    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46"} err="failed to get container status \"d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46\": rpc error: code = NotFound desc = could not find container \"d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46\": container with ID starting with d388501d890b346bdc857275ad10e74e800e13c7d1a92bf2f3276ced7a43ba46 not found: ID does not exist"
Oct 31 20:08:40.807247 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.807168    5006 scope.go:122] "RemoveContainer" containerID="81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d"
Oct 31 20:08:40.820207 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.820183    5006 scope.go:122] "RemoveContainer" containerID="81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d"
Oct 31 20:08:40.820562 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:08:40.820541    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d\": container with ID starting with 81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d not found: ID does not exist" containerID="81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d"
Oct 31 20:08:40.820619 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.820572    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d"} err="failed to get container status \"81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d\": rpc error: code = NotFound desc = could not find container \"81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d\": container with ID starting with 81ec9ac6dd2022c689ce81949ee004f59cc347152c29528403a8ed8d46ed880d not found: ID does not exist"
Oct 31 20:08:40.960927 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.960893    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"]
Oct 31 20:08:40.963679 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:40.963645    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-662be1-695d8995cc-xknk9"]
Oct 31 20:08:41.121217 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:41.121180    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"]
Oct 31 20:08:41.124295 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:41.124273    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-662be1-5c57c7477f-2srzn"]
Oct 31 20:08:41.599285 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:41.599249    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d0fa26-cffd-44a1-b12a-ba9d6207d3d7" path="/var/lib/kubelet/pods/33d0fa26-cffd-44a1-b12a-ba9d6207d3d7/volumes"
Oct 31 20:08:41.599661 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:41.599632    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ad6ab8-45f3-4ae4-88f8-7572147cabb7" path="/var/lib/kubelet/pods/e1ad6ab8-45f3-4ae4-88f8-7572147cabb7/volumes"
Oct 31 20:08:43.765881 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:43.765844    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:45.623721 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:45.623628    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:45.843663 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:45.843621    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:46.058568 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:46.058469    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:48.676023 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:48.675985    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:51.831288 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:51.831252    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:52.069043 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:52.069011    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:53.588900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:53.588864    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:58.499892 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:08:58.499852    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:03.382117 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:03.382076    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:05.394078 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:09:05.394039    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:09:08.300493 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:08.300432    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:09.280032 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:09.279987    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:09:14.877813 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:14.877781    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:18.204403 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:18.204361    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:22.642193 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:22.642154    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:24.226178 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:24.226143    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:24.511880 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:24.511796    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:24.564851 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:24.564819    5006 scope.go:122] "RemoveContainer" containerID="5a9266902e7dac8db67c5bfb866ea585d44bc0422a2a7c84ae74ab4d102379a1"
Oct 31 20:09:24.575164 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:24.575143    5006 scope.go:122] "RemoveContainer" containerID="c04b72ec16b2e920d391dca71c50f1a446af49cf5fc94fb129231da8830bb134"
Oct 31 20:09:24.584783 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:24.584762    5006 scope.go:122] "RemoveContainer" containerID="07806075a9806701b0f0ecb9958ad78da8e244305ba8caf569ca321e8e5b465d"
Oct 31 20:09:24.669270 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:24.669228    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"]
Oct 31 20:09:24.719064 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:24.719031    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-662be1-ch8sc"]
Oct 31 20:09:24.755478 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:24.755423    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:25.119141 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:25.119096    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:25.598780 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:25.598745    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3849823f-6e58-43d4-944c-ba9d25160061" path="/var/lib/kubelet/pods/3849823f-6e58-43d4-944c-ba9d25160061/volumes"
Oct 31 20:09:29.693722 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:29.693687    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:29.930295 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:29.930249    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:31.270889 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:31.270788    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:31.505277 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:31.505238    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"]
Oct 31 20:09:31.505543 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:31.505502    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy" containerID="cri-o://d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4" gracePeriod=30
Oct 31 20:09:31.543038 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:31.542929    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:31.797978 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:31.797886    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:34.809605 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:34.809570    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:35.393924 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:09:35.393884    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:09:36.403790 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.403746    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:09:36.788726 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.788690    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 20:09:36.955730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.955623    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-kube-api-access-5rdrv\" (UniqueName: \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-kube-api-access-5rdrv\") pod \"fc592dbb-a347-4476-a4c3-17987158ffa2\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") "
Oct 31 20:09:36.955730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.955703    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-socket\") pod \"fc592dbb-a347-4476-a4c3-17987158ffa2\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") "
Oct 31 20:09:36.955955 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.955820    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-credential-socket\") pod \"fc592dbb-a347-4476-a4c3-17987158ffa2\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") "
Oct 31 20:09:36.955955 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.955881    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-podinfo\") pod \"fc592dbb-a347-4476-a4c3-17987158ffa2\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") "
Oct 31 20:09:36.956291 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.956258    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-envoy\") pod \"fc592dbb-a347-4476-a4c3-17987158ffa2\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") "
Oct 31 20:09:36.956291 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.956012    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-socket" pod "fc592dbb-a347-4476-a4c3-17987158ffa2" (UID: "fc592dbb-a347-4476-a4c3-17987158ffa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:09:36.956440 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.956293    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc592dbb-a347-4476-a4c3-17987158ffa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc592dbb-a347-4476-a4c3-17987158ffa2-istiod-ca-cert\") pod \"fc592dbb-a347-4476-a4c3-17987158ffa2\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") "
Oct 31 20:09:36.956440 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.956356    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-token\" (UniqueName: \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-token\") pod \"fc592dbb-a347-4476-a4c3-17987158ffa2\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") "
Oct 31 20:09:36.956440 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.956100    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-credential-socket" pod "fc592dbb-a347-4476-a4c3-17987158ffa2" (UID: "fc592dbb-a347-4476-a4c3-17987158ffa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:09:36.956440 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.956394    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-data\") pod \"fc592dbb-a347-4476-a4c3-17987158ffa2\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") "
Oct 31 20:09:36.956712 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.956537    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-certs\") pod \"fc592dbb-a347-4476-a4c3-17987158ffa2\" (UID: \"fc592dbb-a347-4476-a4c3-17987158ffa2\") "
Oct 31 20:09:36.956712 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.956639    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc592dbb-a347-4476-a4c3-17987158ffa2-istiod-ca-cert" pod "fc592dbb-a347-4476-a4c3-17987158ffa2" (UID: "fc592dbb-a347-4476-a4c3-17987158ffa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:09:36.957037 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.957010    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-data" pod "fc592dbb-a347-4476-a4c3-17987158ffa2" (UID: "fc592dbb-a347-4476-a4c3-17987158ffa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:09:36.957037 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.957027    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-certs" pod "fc592dbb-a347-4476-a4c3-17987158ffa2" (UID: "fc592dbb-a347-4476-a4c3-17987158ffa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:09:36.957259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.957015    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:36.957259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.957064    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:36.957259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.957082    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc592dbb-a347-4476-a4c3-17987158ffa2-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:36.958837 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.958808    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-podinfo" pod "fc592dbb-a347-4476-a4c3-17987158ffa2" (UID: "fc592dbb-a347-4476-a4c3-17987158ffa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:09:36.958958 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.958842    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-envoy" pod "fc592dbb-a347-4476-a4c3-17987158ffa2" (UID: "fc592dbb-a347-4476-a4c3-17987158ffa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:09:36.959215 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.959186    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-kube-api-access-5rdrv" pod "fc592dbb-a347-4476-a4c3-17987158ffa2" (UID: "fc592dbb-a347-4476-a4c3-17987158ffa2"). InnerVolumeSpecName "kube-api-access-5rdrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:09:36.959492 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:36.959472    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-token" pod "fc592dbb-a347-4476-a4c3-17987158ffa2" (UID: "fc592dbb-a347-4476-a4c3-17987158ffa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:09:37.058321 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.058275    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:37.058321 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.058313    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rdrv\" (UniqueName: \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-kube-api-access-5rdrv\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:37.058321 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.058327    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:37.058598 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.058340    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:37.058598 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.058351    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:37.058598 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.058363    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc592dbb-a347-4476-a4c3-17987158ffa2-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:37.191048 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.191007    5006 generic.go:356] "Generic (PLEG): container finished" podID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerID="d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4" exitCode=0
Oct 31 20:09:37.191241 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.191057    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"
Oct 31 20:09:37.191241 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.191088    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" event={"ID":"fc592dbb-a347-4476-a4c3-17987158ffa2","Type":"ContainerDied","Data":"d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4"}
Oct 31 20:09:37.191241 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.191144    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7" event={"ID":"fc592dbb-a347-4476-a4c3-17987158ffa2","Type":"ContainerDied","Data":"4da872d131c6c358a1b1d1eb8d44f92748555f52e9a0a5f400bc55a1253bcdd8"}
Oct 31 20:09:37.191241 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.191165    5006 scope.go:122] "RemoveContainer" containerID="d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4"
Oct 31 20:09:37.202299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.202277    5006 scope.go:122] "RemoveContainer" containerID="d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4"
Oct 31 20:09:37.202639 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:09:37.202618    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4\": container with ID starting with d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4 not found: ID does not exist" containerID="d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4"
Oct 31 20:09:37.202748 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.202661    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4"} err="failed to get container status \"d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4\": rpc error: code = NotFound desc = could not find container \"d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4\": container with ID starting with d7cb729a017d14d5e840841f2fd82c4668031b5d6cefda1e40f99a52886590b4 not found: ID does not exist"
Oct 31 20:09:37.359498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.359459    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"]
Oct 31 20:09:37.364725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.364696    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-566j7"]
Oct 31 20:09:37.598467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:37.598424    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" path="/var/lib/kubelet/pods/fc592dbb-a347-4476-a4c3-17987158ffa2/volumes"
Oct 31 20:09:39.703049 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:39.703014    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:44.580868 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:44.580802    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:49.514424 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:49.514374    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:54.483230 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:54.483196    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:56.051126 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:56.051083    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"]
Oct 31 20:09:56.101234 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:56.101200    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ae198e-72cjh"]
Oct 31 20:09:57.600485 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:57.600444    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91109a4-d18d-483b-872a-ae49dc4d580a" path="/var/lib/kubelet/pods/b91109a4-d18d-483b-872a-ae49dc4d580a/volumes"
Oct 31 20:09:59.547663 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:09:59.547615    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:03.815524 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.815486    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"]
Oct 31 20:10:03.816051 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816024    5006 cpu_manager.go:414] "containerMap: removing container" podUID="33d0fa26-cffd-44a1-b12a-ba9d6207d3d7" containerName="payload-processing"
Oct 31 20:10:03.816051 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816038    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d0fa26-cffd-44a1-b12a-ba9d6207d3d7" containerName="payload-processing"
Oct 31 20:10:03.816051 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816047    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e1ad6ab8-45f3-4ae4-88f8-7572147cabb7" containerName="payload-pre-processing"
Oct 31 20:10:03.816051 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816052    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ad6ab8-45f3-4ae4-88f8-7572147cabb7" containerName="payload-pre-processing"
Oct 31 20:10:03.816195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816069    5006 cpu_manager.go:414] "containerMap: removing container" podUID="52a1c6f3-45cd-4d90-8429-67fa43199198" containerName="revoke-keys"
Oct 31 20:10:03.816195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816075    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a1c6f3-45cd-4d90-8429-67fa43199198" containerName="revoke-keys"
Oct 31 20:10:03.816195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816095    5006 cpu_manager.go:414] "containerMap: removing container" podUID="9327afce-2584-4b26-97a8-85cc442e6bc2" containerName="revoke-keys"
Oct 31 20:10:03.816195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816100    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9327afce-2584-4b26-97a8-85cc442e6bc2" containerName="revoke-keys"
Oct 31 20:10:03.816195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816109    5006 cpu_manager.go:414] "containerMap: removing container" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy"
Oct 31 20:10:03.816195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816113    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy"
Oct 31 20:10:03.816195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816183    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d0fa26-cffd-44a1-b12a-ba9d6207d3d7" containerName="payload-processing"
Oct 31 20:10:03.816195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816190    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ad6ab8-45f3-4ae4-88f8-7572147cabb7" containerName="payload-pre-processing"
Oct 31 20:10:03.816451 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816199    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a1c6f3-45cd-4d90-8429-67fa43199198" containerName="revoke-keys"
Oct 31 20:10:03.816451 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816210    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9327afce-2584-4b26-97a8-85cc442e6bc2" containerName="revoke-keys"
Oct 31 20:10:03.816451 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.816217    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc592dbb-a347-4476-a4c3-17987158ffa2" containerName="istio-proxy"
Oct 31 20:10:03.819783 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.819763    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:03.875530 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.875492    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:10:03.924402 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.924365    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:10:03.924402 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.924396    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:10:03.924615 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.924361    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:10:03.927450 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.927424    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"]
Oct 31 20:10:03.952133 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.952097    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/b585104a-8928-416a-a03f-f612804c8fa5-kube-api-access-swvvb\") pod \"maas-api-revoke-sub-52950dab-zmwbz\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:03.952345 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.952156    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52950dab-zmwbz\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:03.952399 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:03.952362    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52950dab-zmwbz\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:04.053935 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:04.053892    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52950dab-zmwbz\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:04.054128 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:04.054019    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52950dab-zmwbz\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:04.054128 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:04.054058    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/b585104a-8928-416a-a03f-f612804c8fa5-kube-api-access-swvvb\") pod \"maas-api-revoke-sub-52950dab-zmwbz\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:04.054779 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:04.054756    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52950dab-zmwbz\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:04.056355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:04.056337    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52950dab-zmwbz\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:04.114429 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:04.114398    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/b585104a-8928-416a-a03f-f612804c8fa5-kube-api-access-swvvb\") pod \"maas-api-revoke-sub-52950dab-zmwbz\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:04.133306 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:04.133261    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:04.409090 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:04.409008    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:04.505049 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:10:04.504967    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb585104a_8928_416a_a03f_f612804c8fa5.slice/crio-29e40cd81cdcc70c5bbc32fa4635d51eb4ed6d4743e8f23783ff48338d9a0cf8 WatchSource:0}: Error finding container 29e40cd81cdcc70c5bbc32fa4635d51eb4ed6d4743e8f23783ff48338d9a0cf8: Status 404 returned error can't find the container with id 29e40cd81cdcc70c5bbc32fa4635d51eb4ed6d4743e8f23783ff48338d9a0cf8
Oct 31 20:10:04.508417 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:04.508395    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"]
Oct 31 20:10:05.389617 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:05.389578    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz" event={"ID":"b585104a-8928-416a-a03f-f612804c8fa5","Type":"ContainerStarted","Data":"def9219d43af5e944ffd0f3e0a0363df20434a56afec29971a7bd2f12caae483"}
Oct 31 20:10:05.390075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:05.389633    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz" event={"ID":"b585104a-8928-416a-a03f-f612804c8fa5","Type":"ContainerStarted","Data":"29e40cd81cdcc70c5bbc32fa4635d51eb4ed6d4743e8f23783ff48338d9a0cf8"}
Oct 31 20:10:05.394767 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:10:05.394731    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:10:05.519546 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:05.519470    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz" podStartSLOduration=2.519454168 podStartE2EDuration="2.519454168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:10:05.517541431 +0000 UTC m=+3224.047075466" watchObservedRunningTime="2026-10-31 20:10:05.519454168 +0000 UTC m=+3224.048988220"
Oct 31 20:10:06.397432 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:06.397391    5006 generic.go:356] "Generic (PLEG): container finished" podID="b585104a-8928-416a-a03f-f612804c8fa5" containerID="def9219d43af5e944ffd0f3e0a0363df20434a56afec29971a7bd2f12caae483" exitCode=0
Oct 31 20:10:06.397911 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:06.397472    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz" event={"ID":"b585104a-8928-416a-a03f-f612804c8fa5","Type":"ContainerDied","Data":"def9219d43af5e944ffd0f3e0a0363df20434a56afec29971a7bd2f12caae483"}
Oct 31 20:10:07.555013 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:07.554982    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:07.696340 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:07.696232    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b585104a-8928-416a-a03f-f612804c8fa5-kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/b585104a-8928-416a-a03f-f612804c8fa5-kube-api-access-swvvb\") pod \"b585104a-8928-416a-a03f-f612804c8fa5\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") "
Oct 31 20:10:07.696509 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:07.696406    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-openshift-ca\") pod \"b585104a-8928-416a-a03f-f612804c8fa5\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") "
Oct 31 20:10:07.696509 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:07.696468    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-xks-ca\") pod \"b585104a-8928-416a-a03f-f612804c8fa5\" (UID: \"b585104a-8928-416a-a03f-f612804c8fa5\") "
Oct 31 20:10:07.696900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:07.696863    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-openshift-ca" pod "b585104a-8928-416a-a03f-f612804c8fa5" (UID: "b585104a-8928-416a-a03f-f612804c8fa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:10:07.698715 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:07.698676    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b585104a-8928-416a-a03f-f612804c8fa5-kube-api-access-swvvb" pod "b585104a-8928-416a-a03f-f612804c8fa5" (UID: "b585104a-8928-416a-a03f-f612804c8fa5"). InnerVolumeSpecName "kube-api-access-swvvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:10:07.699115 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:07.699088    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-xks-ca" pod "b585104a-8928-416a-a03f-f612804c8fa5" (UID: "b585104a-8928-416a-a03f-f612804c8fa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:10:07.797528 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:07.797481    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:07.797528 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:07.797516    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b585104a-8928-416a-a03f-f612804c8fa5-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:07.797528 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:07.797535    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/b585104a-8928-416a-a03f-f612804c8fa5-kube-api-access-swvvb\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:08.411090 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:08.411060    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"
Oct 31 20:10:08.411312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:08.411090    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz" event={"ID":"b585104a-8928-416a-a03f-f612804c8fa5","Type":"ContainerDied","Data":"29e40cd81cdcc70c5bbc32fa4635d51eb4ed6d4743e8f23783ff48338d9a0cf8"}
Oct 31 20:10:08.411312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:08.411135    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e40cd81cdcc70c5bbc32fa4635d51eb4ed6d4743e8f23783ff48338d9a0cf8"
Oct 31 20:10:09.371963 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:09.371929    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:14.286633 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:14.286598    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:15.881342 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:15.881301    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:16.833715 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.833680    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"]
Oct 31 20:10:16.834215 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.834201    5006 cpu_manager.go:414] "containerMap: removing container" podUID="b585104a-8928-416a-a03f-f612804c8fa5" containerName="revoke-keys"
Oct 31 20:10:16.834263 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.834215    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b585104a-8928-416a-a03f-f612804c8fa5" containerName="revoke-keys"
Oct 31 20:10:16.834308 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.834299    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b585104a-8928-416a-a03f-f612804c8fa5" containerName="revoke-keys"
Oct 31 20:10:16.837598 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.837578    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:16.891078 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.891040    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5tn\" (UniqueName: \"kubernetes.io/projected/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-kube-api-access-mg5tn\") pod \"maas-api-revoke-sub-c343631f-hl6qj\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:16.891561 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.891173    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c343631f-hl6qj\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:16.891561 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.891293    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c343631f-hl6qj\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:16.892801 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.892774    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"]
Oct 31 20:10:16.939924 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.939885    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:10:16.939924 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.939920    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:10:16.941019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.940983    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:10:16.941309 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.941292    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:10:16.992210 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.992165    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c343631f-hl6qj\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:16.992408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.992272    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5tn\" (UniqueName: \"kubernetes.io/projected/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-kube-api-access-mg5tn\") pod \"maas-api-revoke-sub-c343631f-hl6qj\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:16.992408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.992369    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c343631f-hl6qj\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:16.992999 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.992968    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c343631f-hl6qj\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:16.994996 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:16.994964    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c343631f-hl6qj\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:17.050195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:17.050152    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5tn\" (UniqueName: \"kubernetes.io/projected/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-kube-api-access-mg5tn\") pod \"maas-api-revoke-sub-c343631f-hl6qj\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:17.151755 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:17.151711    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:17.524985 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:10:17.524948    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6baae907_f3cd_43dd_ba8a_1d9e6776b4de.slice/crio-1929c44935ef99160a550ff5486278347279db46e1a3f703b18484308348e8bb WatchSource:0}: Error finding container 1929c44935ef99160a550ff5486278347279db46e1a3f703b18484308348e8bb: Status 404 returned error can't find the container with id 1929c44935ef99160a550ff5486278347279db46e1a3f703b18484308348e8bb
Oct 31 20:10:17.529214 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:17.529135    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"]
Oct 31 20:10:18.478539 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:18.478504    5006 generic.go:356] "Generic (PLEG): container finished" podID="6baae907-f3cd-43dd-ba8a-1d9e6776b4de" containerID="655aec9058995280a5043b8c5bc739722c28c99c798b1abd5123035e69809f9d" exitCode=0
Oct 31 20:10:18.478991 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:18.478594    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj" event={"ID":"6baae907-f3cd-43dd-ba8a-1d9e6776b4de","Type":"ContainerDied","Data":"655aec9058995280a5043b8c5bc739722c28c99c798b1abd5123035e69809f9d"}
Oct 31 20:10:18.478991 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:18.478673    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj" event={"ID":"6baae907-f3cd-43dd-ba8a-1d9e6776b4de","Type":"ContainerStarted","Data":"1929c44935ef99160a550ff5486278347279db46e1a3f703b18484308348e8bb"}
Oct 31 20:10:19.231624 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.231586    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:19.648307 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.648274    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:19.720924 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.720877    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-openshift-ca\") pod \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") "
Oct 31 20:10:19.721114 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.720987    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-kube-api-access-mg5tn\" (UniqueName: \"kubernetes.io/projected/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-kube-api-access-mg5tn\") pod \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") "
Oct 31 20:10:19.721114 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.721050    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-xks-ca\") pod \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\" (UID: \"6baae907-f3cd-43dd-ba8a-1d9e6776b4de\") "
Oct 31 20:10:19.721282 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.721242    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-openshift-ca" pod "6baae907-f3cd-43dd-ba8a-1d9e6776b4de" (UID: "6baae907-f3cd-43dd-ba8a-1d9e6776b4de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:10:19.721472 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.721451    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:19.723389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.723352    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-kube-api-access-mg5tn" pod "6baae907-f3cd-43dd-ba8a-1d9e6776b4de" (UID: "6baae907-f3cd-43dd-ba8a-1d9e6776b4de"). InnerVolumeSpecName "kube-api-access-mg5tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:10:19.723389 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.723366    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-xks-ca" pod "6baae907-f3cd-43dd-ba8a-1d9e6776b4de" (UID: "6baae907-f3cd-43dd-ba8a-1d9e6776b4de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:10:19.823029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.822932    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5tn\" (UniqueName: \"kubernetes.io/projected/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-kube-api-access-mg5tn\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:19.823029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:19.822969    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6baae907-f3cd-43dd-ba8a-1d9e6776b4de-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:20.492687 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:20.492633    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"
Oct 31 20:10:20.492903 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:20.492633    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj" event={"ID":"6baae907-f3cd-43dd-ba8a-1d9e6776b4de","Type":"ContainerDied","Data":"1929c44935ef99160a550ff5486278347279db46e1a3f703b18484308348e8bb"}
Oct 31 20:10:20.492903 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:20.492767    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1929c44935ef99160a550ff5486278347279db46e1a3f703b18484308348e8bb"
Oct 31 20:10:24.094353 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:24.094314    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:24.627504 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:24.627470    5006 scope.go:122] "RemoveContainer" containerID="9528e66a4eac064f0171e3b99a9f5f9b2ff3d605feb58ae8acff98798147820e"
Oct 31 20:10:24.887724 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:24.887607    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:10:29.065422 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:29.065387    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:29.980817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:29.980783    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:33.872605 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:33.872568    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:35.393610 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:10:35.393569    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:10:36.052927 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:36.052894    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"]
Oct 31 20:10:36.076540 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:36.076503    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"]
Oct 31 20:10:36.103208 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:36.103176    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff718e85-l5smr"]
Oct 31 20:10:36.105817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:36.105793    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e49e9f-cbv77"]
Oct 31 20:10:37.599900 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:37.599866    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa05dcc-89ae-4259-8e79-31b57d63653e" path="/var/lib/kubelet/pods/6aa05dcc-89ae-4259-8e79-31b57d63653e/volumes"
Oct 31 20:10:37.600323 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:37.600308    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a44c69-e1b4-4bab-aafd-ef66056cbed1" path="/var/lib/kubelet/pods/91a44c69-e1b4-4bab-aafd-ef66056cbed1/volumes"
Oct 31 20:10:38.864722 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:38.864691    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:43.716765 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:43.716730    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:45.687382 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:45.687342    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:10:48.655094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:48.655064    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:52.062231 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:52.062191    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"]
Oct 31 20:10:52.112740 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:52.112699    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"]
Oct 31 20:10:52.115097 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:52.115072    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5776c90d-wcfsd"]
Oct 31 20:10:52.117221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:52.117199    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f7d4cb1-52gz2"]
Oct 31 20:10:53.598847 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:53.598813    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137496ba-aa62-42b7-92a4-e802faf99afc" path="/var/lib/kubelet/pods/137496ba-aa62-42b7-92a4-e802faf99afc/volumes"
Oct 31 20:10:53.599288 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:53.599274    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe0d09d-180f-40fd-ad90-21a65f5245c8" path="/var/lib/kubelet/pods/efe0d09d-180f-40fd-ad90-21a65f5245c8/volumes"
Oct 31 20:10:53.660888 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:53.660855    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:55.418066 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:55.418025    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:55.916840 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:55.916798    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"]
Oct 31 20:10:55.917600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:55.917578    5006 cpu_manager.go:414] "containerMap: removing container" podUID="6baae907-f3cd-43dd-ba8a-1d9e6776b4de" containerName="revoke-keys"
Oct 31 20:10:55.917675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:55.917601    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6baae907-f3cd-43dd-ba8a-1d9e6776b4de" containerName="revoke-keys"
Oct 31 20:10:55.917721 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:55.917714    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6baae907-f3cd-43dd-ba8a-1d9e6776b4de" containerName="revoke-keys"
Oct 31 20:10:55.920910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:55.920887    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:55.976051 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:55.976017    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"]
Oct 31 20:10:56.018202 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.018165    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:56.018202 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.018211    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:56.018438 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.018335    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpvr\" (UniqueName: \"kubernetes.io/projected/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-kube-api-access-qbpvr\") pod \"maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:56.023111 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.023075    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:10:56.023111 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.023094    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:10:56.023293 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.023108    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:10:56.024885 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.024868    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:10:56.119450 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.119404    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpvr\" (UniqueName: \"kubernetes.io/projected/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-kube-api-access-qbpvr\") pod \"maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:56.119700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.119525    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:56.119700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.119553    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:56.120355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.120334    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:56.122052 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.122032    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:56.177824 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.177747    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpvr\" (UniqueName: \"kubernetes.io/projected/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-kube-api-access-qbpvr\") pod \"maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:56.235761 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.235713    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:56.611858 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:10:56.611822    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b94fe5_1fb8_4f08_b7bf_0e204ce2d202.slice/crio-6e9df1a2e8ebbdf703dc1813ddd0bfa9026dd361f90f0a9f42e9d3b5c6491ae4 WatchSource:0}: Error finding container 6e9df1a2e8ebbdf703dc1813ddd0bfa9026dd361f90f0a9f42e9d3b5c6491ae4: Status 404 returned error can't find the container with id 6e9df1a2e8ebbdf703dc1813ddd0bfa9026dd361f90f0a9f42e9d3b5c6491ae4
Oct 31 20:10:56.614436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.614408    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"]
Oct 31 20:10:56.733159 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:56.733125    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm" event={"ID":"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202","Type":"ContainerStarted","Data":"6e9df1a2e8ebbdf703dc1813ddd0bfa9026dd361f90f0a9f42e9d3b5c6491ae4"}
Oct 31 20:10:57.741415 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:57.741377    5006 generic.go:356] "Generic (PLEG): container finished" podID="e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202" containerID="4d6a670629234602861d0f13fd0472851c4ffe9dd5b74484ab8140e7b678e6e7" exitCode=0
Oct 31 20:10:57.741865 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:57.741466    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm" event={"ID":"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202","Type":"ContainerDied","Data":"4d6a670629234602861d0f13fd0472851c4ffe9dd5b74484ab8140e7b678e6e7"}
Oct 31 20:10:58.757997 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:58.757942    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:58.905139 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:58.905110    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:58.952334 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:58.952295    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-kube-api-access-qbpvr\" (UniqueName: \"kubernetes.io/projected/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-kube-api-access-qbpvr\") pod \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") "
Oct 31 20:10:58.952535 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:58.952516    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-xks-ca\") pod \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") "
Oct 31 20:10:58.952723 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:58.952706    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-openshift-ca\") pod \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\" (UID: \"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202\") "
Oct 31 20:10:58.953041 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:58.953020    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-openshift-ca" pod "e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202" (UID: "e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:10:58.954511 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:58.954490    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-kube-api-access-qbpvr" pod "e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202" (UID: "e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202"). InnerVolumeSpecName "kube-api-access-qbpvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:10:58.954596 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:58.954580    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-xks-ca" pod "e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202" (UID: "e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:10:59.054319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:59.054231    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:59.054319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:59.054265    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:59.054319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:59.054277    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbpvr\" (UniqueName: \"kubernetes.io/projected/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202-kube-api-access-qbpvr\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:59.754926 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:59.754893    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"
Oct 31 20:10:59.755121 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:59.754933    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm" event={"ID":"e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202","Type":"ContainerDied","Data":"6e9df1a2e8ebbdf703dc1813ddd0bfa9026dd361f90f0a9f42e9d3b5c6491ae4"}
Oct 31 20:10:59.755121 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:10:59.754990    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9df1a2e8ebbdf703dc1813ddd0bfa9026dd361f90f0a9f42e9d3b5c6491ae4"
Oct 31 20:11:01.975307 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:01.975214    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"]
Oct 31 20:11:01.975763 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:01.975738    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202" containerName="revoke-keys"
Oct 31 20:11:01.975763 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:01.975752    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202" containerName="revoke-keys"
Oct 31 20:11:01.975875 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:01.975864    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202" containerName="revoke-keys"
Oct 31 20:11:01.979318 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:01.979297    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.032044 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.032013    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"]
Oct 31 20:11:02.079631 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.079593    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:11:02.079631 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.079616    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:11:02.079631 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.079637    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:11:02.082207 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.082188    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:11:02.089741 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.089716    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb219bea-4vc4c\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.089869 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.089746    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb219bea-4vc4c\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.090010 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.089989    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddpk\" (UniqueName: \"kubernetes.io/projected/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-kube-api-access-9ddpk\") pod \"maas-api-revoke-sub-fb219bea-4vc4c\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.190940 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.190907    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddpk\" (UniqueName: \"kubernetes.io/projected/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-kube-api-access-9ddpk\") pod \"maas-api-revoke-sub-fb219bea-4vc4c\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.191171 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.190959    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb219bea-4vc4c\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.191171 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.191129    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb219bea-4vc4c\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.191551 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.191533    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb219bea-4vc4c\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.193424 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.193406    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb219bea-4vc4c\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.250008 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.249929    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddpk\" (UniqueName: \"kubernetes.io/projected/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-kube-api-access-9ddpk\") pod \"maas-api-revoke-sub-fb219bea-4vc4c\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.292908 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.292874    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:02.664929 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:11:02.664899    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154c5972_4b1d_4d5d_9e70_fd2575cde1ca.slice/crio-2146f24804723a32f1f030f0868e7c3b2435a3b33a55e60e1313e6c1a491a7f5 WatchSource:0}: Error finding container 2146f24804723a32f1f030f0868e7c3b2435a3b33a55e60e1313e6c1a491a7f5: Status 404 returned error can't find the container with id 2146f24804723a32f1f030f0868e7c3b2435a3b33a55e60e1313e6c1a491a7f5
Oct 31 20:11:02.667125 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.667100    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"]
Oct 31 20:11:02.774282 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:02.774242    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c" event={"ID":"154c5972-4b1d-4d5d-9e70-fd2575cde1ca","Type":"ContainerStarted","Data":"2146f24804723a32f1f030f0868e7c3b2435a3b33a55e60e1313e6c1a491a7f5"}
Oct 31 20:11:03.656237 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:03.656187    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:11:03.782544 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:03.782458    5006 generic.go:356] "Generic (PLEG): container finished" podID="154c5972-4b1d-4d5d-9e70-fd2575cde1ca" containerID="dbf6b5a7a5295b6b1ebdf339785cd9ba407bda37f72820b86f5ea399c042d4ed" exitCode=0
Oct 31 20:11:03.782716 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:03.782550    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c" event={"ID":"154c5972-4b1d-4d5d-9e70-fd2575cde1ca","Type":"ContainerDied","Data":"dbf6b5a7a5295b6b1ebdf339785cd9ba407bda37f72820b86f5ea399c042d4ed"}
Oct 31 20:11:04.005620 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:04.005577    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:04.943846 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:04.943814    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:05.022953 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.022920    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-kube-api-access-9ddpk\" (UniqueName: \"kubernetes.io/projected/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-kube-api-access-9ddpk\") pod \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") "
Oct 31 20:11:05.023164 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.022970    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-xks-ca\") pod \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") "
Oct 31 20:11:05.023164 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.023073    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-openshift-ca\") pod \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\" (UID: \"154c5972-4b1d-4d5d-9e70-fd2575cde1ca\") "
Oct 31 20:11:05.023482 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.023452    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-openshift-ca" pod "154c5972-4b1d-4d5d-9e70-fd2575cde1ca" (UID: "154c5972-4b1d-4d5d-9e70-fd2575cde1ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:11:05.023625 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.023584    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:05.025154 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.025132    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-kube-api-access-9ddpk" pod "154c5972-4b1d-4d5d-9e70-fd2575cde1ca" (UID: "154c5972-4b1d-4d5d-9e70-fd2575cde1ca"). InnerVolumeSpecName "kube-api-access-9ddpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:11:05.025504 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.025477    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-xks-ca" pod "154c5972-4b1d-4d5d-9e70-fd2575cde1ca" (UID: "154c5972-4b1d-4d5d-9e70-fd2575cde1ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:11:05.124557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.124517    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ddpk\" (UniqueName: \"kubernetes.io/projected/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-kube-api-access-9ddpk\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:05.124557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.124557    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/154c5972-4b1d-4d5d-9e70-fd2575cde1ca-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:05.394356 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:11:05.394271    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:11:05.796364 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.796283    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"
Oct 31 20:11:05.796530 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.796284    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c" event={"ID":"154c5972-4b1d-4d5d-9e70-fd2575cde1ca","Type":"ContainerDied","Data":"2146f24804723a32f1f030f0868e7c3b2435a3b33a55e60e1313e6c1a491a7f5"}
Oct 31 20:11:05.796530 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:05.796415    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2146f24804723a32f1f030f0868e7c3b2435a3b33a55e60e1313e6c1a491a7f5"
Oct 31 20:11:07.430285 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:07.430248    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"]
Oct 31 20:11:07.539458 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:07.539419    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"]
Oct 31 20:11:07.872337 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:07.872299    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:08.085971 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:08.085939    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:08.997895 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:08.997852    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:12.431108 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:12.431056    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd" podUID="8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6" containerName="payload-processing" containerID="cri-o://236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e" gracePeriod=25
Oct 31 20:11:12.539842 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:12.539776    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj" podUID="e7330358-d4c3-4f17-a61c-d794d4425a28" containerName="payload-pre-processing" containerID="cri-o://6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee" gracePeriod=25
Oct 31 20:11:13.886761 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:13.886714    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:18.822390 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:18.822344    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:22.243538 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:22.243511    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:11:22.320108 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:22.320081    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:11:22.362039 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:22.361993    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd" podUID="8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:11:22.362234 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:22.362044    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj" podUID="e7330358-d4c3-4f17-a61c-d794d4425a28" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:11:23.210574 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:23.210535    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:11:23.775704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:23.775618    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:24.660222 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:24.660191    5006 scope.go:122] "RemoveContainer" containerID="7a39620afd7c8cf573498c68a3048f1358f3404b5b1fca78b8af7e102c40199b"
Oct 31 20:11:24.669600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:24.669579    5006 scope.go:122] "RemoveContainer" containerID="cc60ef7e357a3e0401aa4b55873b41ed6148c6ae92941842c42ca8cc5dda6561"
Oct 31 20:11:24.678391 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:24.678371    5006 scope.go:122] "RemoveContainer" containerID="72a78bbaa0a35d9367bac0934d3eeb5bf969aedc03182f224f4c47efffd19f93"
Oct 31 20:11:24.687385 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:24.687365    5006 scope.go:122] "RemoveContainer" containerID="453bf9dcb3bff573c841f23c4d4de78226619d02bb762d350168edb3780e8d79"
Oct 31 20:11:28.657250 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:28.657204    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:29.416725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:29.416689    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:30.310845 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.310808    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"]
Oct 31 20:11:30.311400 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.311385    5006 cpu_manager.go:414] "containerMap: removing container" podUID="154c5972-4b1d-4d5d-9e70-fd2575cde1ca" containerName="revoke-keys"
Oct 31 20:11:30.311456 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.311400    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154c5972-4b1d-4d5d-9e70-fd2575cde1ca" containerName="revoke-keys"
Oct 31 20:11:30.311828 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.311506    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="154c5972-4b1d-4d5d-9e70-fd2575cde1ca" containerName="revoke-keys"
Oct 31 20:11:30.316698 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.316676    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:30.379824 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.379787    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:11:30.379824 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.379822    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:11:30.380057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.379851    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:11:30.380057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.379876    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:11:30.389584 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.389545    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"]
Oct 31 20:11:30.396206 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.396156    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmktm\" (UniqueName: \"kubernetes.io/projected/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-kube-api-access-rmktm\") pod \"maas-api-revoke-sub-0e242190-2l58k\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:30.396206 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.396201    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e242190-2l58k\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:30.396460 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.396232    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e242190-2l58k\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:30.465868 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.465798    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:30.497079 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.497044    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmktm\" (UniqueName: \"kubernetes.io/projected/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-kube-api-access-rmktm\") pod \"maas-api-revoke-sub-0e242190-2l58k\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:30.497298 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.497095    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e242190-2l58k\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:30.497298 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.497131    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e242190-2l58k\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:30.497989 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.497965    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e242190-2l58k\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:30.499757 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.499731    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e242190-2l58k\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:30.555472 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.555438    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmktm\" (UniqueName: \"kubernetes.io/projected/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-kube-api-access-rmktm\") pod \"maas-api-revoke-sub-0e242190-2l58k\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:30.630608 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:30.630574    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:31.005551 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:31.005498    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"]
Oct 31 20:11:31.009299 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:11:31.009258    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e292e4_cc4f_475f_8f95_8c3eb76fb5bc.slice/crio-bc15c655d18921456dd7da1fcb1ed3b18194f15dd5c8088aa12c5f89907860e8 WatchSource:0}: Error finding container bc15c655d18921456dd7da1fcb1ed3b18194f15dd5c8088aa12c5f89907860e8: Status 404 returned error can't find the container with id bc15c655d18921456dd7da1fcb1ed3b18194f15dd5c8088aa12c5f89907860e8
Oct 31 20:11:31.960778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:31.960739    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k" event={"ID":"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc","Type":"ContainerStarted","Data":"43b1a38107fefea3578d9935139a13c5dfabde7c6ddfdf2926e8ad59386006c7"}
Oct 31 20:11:31.960778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:31.960794    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k" event={"ID":"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc","Type":"ContainerStarted","Data":"bc15c655d18921456dd7da1fcb1ed3b18194f15dd5c8088aa12c5f89907860e8"}
Oct 31 20:11:32.076713 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.076638    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k" podStartSLOduration=2.076624437 podStartE2EDuration="2.076624437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:11:32.075298658 +0000 UTC m=+3310.604832729" watchObservedRunningTime="2026-10-31 20:11:32.076624437 +0000 UTC m=+3310.606158489"
Oct 31 20:11:32.362440 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.362396    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj" podUID="e7330358-d4c3-4f17-a61c-d794d4425a28" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:11:32.362630 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.362410    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd" podUID="8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:11:32.714855 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.714822    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:11:32.819023 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.818992    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:11:32.819171 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.818991    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-plugins-config-volume\") pod \"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6\" (UID: \"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6\") "
Oct 31 20:11:32.819171 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.819122    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-kube-api-access-mksdk\" (UniqueName: \"kubernetes.io/projected/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-kube-api-access-mksdk\") pod \"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6\" (UID: \"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6\") "
Oct 31 20:11:32.819349 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.819329    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-plugins-config-volume" pod "8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6" (UID: "8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:11:32.819584 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.819561    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:32.821136 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.821115    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-kube-api-access-mksdk" pod "8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6" (UID: "8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6"). InnerVolumeSpecName "kube-api-access-mksdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:11:32.920199 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.920118    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7330358-d4c3-4f17-a61c-d794d4425a28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7330358-d4c3-4f17-a61c-d794d4425a28-plugins-config-volume\") pod \"e7330358-d4c3-4f17-a61c-d794d4425a28\" (UID: \"e7330358-d4c3-4f17-a61c-d794d4425a28\") "
Oct 31 20:11:32.920353 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.920237    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7330358-d4c3-4f17-a61c-d794d4425a28-kube-api-access-rbd28\" (UniqueName: \"kubernetes.io/projected/e7330358-d4c3-4f17-a61c-d794d4425a28-kube-api-access-rbd28\") pod \"e7330358-d4c3-4f17-a61c-d794d4425a28\" (UID: \"e7330358-d4c3-4f17-a61c-d794d4425a28\") "
Oct 31 20:11:32.920475 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.920462    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mksdk\" (UniqueName: \"kubernetes.io/projected/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6-kube-api-access-mksdk\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:32.920609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.920582    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7330358-d4c3-4f17-a61c-d794d4425a28-plugins-config-volume" pod "e7330358-d4c3-4f17-a61c-d794d4425a28" (UID: "e7330358-d4c3-4f17-a61c-d794d4425a28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:11:32.922348 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.922327    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7330358-d4c3-4f17-a61c-d794d4425a28-kube-api-access-rbd28" pod "e7330358-d4c3-4f17-a61c-d794d4425a28" (UID: "e7330358-d4c3-4f17-a61c-d794d4425a28"). InnerVolumeSpecName "kube-api-access-rbd28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:11:32.969382 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.969340    5006 generic.go:356] "Generic (PLEG): container finished" podID="8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6" containerID="236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e" exitCode=0
Oct 31 20:11:32.969890 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.969427    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"
Oct 31 20:11:32.970031 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.969886    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd" event={"ID":"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6","Type":"ContainerDied","Data":"236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e"}
Oct 31 20:11:32.970031 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.969984    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd" event={"ID":"8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6","Type":"ContainerDied","Data":"837b14d21e53e6a0d121f93363d0c8de3a09aa1d2660cc4317b227cb81289af5"}
Oct 31 20:11:32.970031 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.970021    5006 scope.go:122] "RemoveContainer" containerID="236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e"
Oct 31 20:11:32.971470 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.971443    5006 generic.go:356] "Generic (PLEG): container finished" podID="79e292e4-cc4f-475f-8f95-8c3eb76fb5bc" containerID="43b1a38107fefea3578d9935139a13c5dfabde7c6ddfdf2926e8ad59386006c7" exitCode=0
Oct 31 20:11:32.971611 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.971585    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k" event={"ID":"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc","Type":"ContainerDied","Data":"43b1a38107fefea3578d9935139a13c5dfabde7c6ddfdf2926e8ad59386006c7"}
Oct 31 20:11:32.973132 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.973104    5006 generic.go:356] "Generic (PLEG): container finished" podID="e7330358-d4c3-4f17-a61c-d794d4425a28" containerID="6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee" exitCode=0
Oct 31 20:11:32.973228 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.973174    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj" event={"ID":"e7330358-d4c3-4f17-a61c-d794d4425a28","Type":"ContainerDied","Data":"6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee"}
Oct 31 20:11:32.973228 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.973215    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj" event={"ID":"e7330358-d4c3-4f17-a61c-d794d4425a28","Type":"ContainerDied","Data":"5ebfcb943209444fd2bc8b4b4b9250f519f7eb6724a8ce7ace8e3ed658b42a78"}
Oct 31 20:11:32.973366 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.973276    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"
Oct 31 20:11:32.982848 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.982826    5006 scope.go:122] "RemoveContainer" containerID="236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e"
Oct 31 20:11:32.983197 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:11:32.983160    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e\": container with ID starting with 236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e not found: ID does not exist" containerID="236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e"
Oct 31 20:11:32.983306 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.983208    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e"} err="failed to get container status \"236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e\": rpc error: code = NotFound desc = could not find container \"236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e\": container with ID starting with 236fd808490b401c36957f4d21dc1741a197a891512f224608a1f1ca70056f2e not found: ID does not exist"
Oct 31 20:11:32.983306 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.983234    5006 scope.go:122] "RemoveContainer" containerID="6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee"
Oct 31 20:11:32.994899 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.994875    5006 scope.go:122] "RemoveContainer" containerID="6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee"
Oct 31 20:11:32.995277 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:11:32.995255    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee\": container with ID starting with 6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee not found: ID does not exist" containerID="6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee"
Oct 31 20:11:32.995352 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:32.995289    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee"} err="failed to get container status \"6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee\": rpc error: code = NotFound desc = could not find container \"6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee\": container with ID starting with 6d9a649f7b3d12e3ec2432000579a5301e511170e5abcce22de02feb930a0eee not found: ID does not exist"
Oct 31 20:11:33.021981 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:33.021946    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbd28\" (UniqueName: \"kubernetes.io/projected/e7330358-d4c3-4f17-a61c-d794d4425a28-kube-api-access-rbd28\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:33.021981 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:33.021977    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7330358-d4c3-4f17-a61c-d794d4425a28-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:33.138364 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:33.138318    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"]
Oct 31 20:11:33.142792 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:33.142756    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fd95d7-6cd59485d7-bkhdd"]
Oct 31 20:11:33.253133 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:33.253039    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"]
Oct 31 20:11:33.255606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:33.255584    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fd95d7-78679b88c-8nwzj"]
Oct 31 20:11:33.598738 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:33.598703    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6" path="/var/lib/kubelet/pods/8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6/volumes"
Oct 31 20:11:33.599116 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:33.599103    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7330358-d4c3-4f17-a61c-d794d4425a28" path="/var/lib/kubelet/pods/e7330358-d4c3-4f17-a61c-d794d4425a28/volumes"
Oct 31 20:11:33.677264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:33.677224    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:34.136229 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.136200    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:34.235617 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.235534    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-openshift-ca\") pod \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") "
Oct 31 20:11:34.235780 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.235687    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-xks-ca\") pod \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") "
Oct 31 20:11:34.235780 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.235748    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-kube-api-access-rmktm\" (UniqueName: \"kubernetes.io/projected/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-kube-api-access-rmktm\") pod \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\" (UID: \"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc\") "
Oct 31 20:11:34.236003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.235981    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-openshift-ca" pod "79e292e4-cc4f-475f-8f95-8c3eb76fb5bc" (UID: "79e292e4-cc4f-475f-8f95-8c3eb76fb5bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:11:34.236177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.236163    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:34.237878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.237850    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-xks-ca" pod "79e292e4-cc4f-475f-8f95-8c3eb76fb5bc" (UID: "79e292e4-cc4f-475f-8f95-8c3eb76fb5bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:11:34.238006 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.237885    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-kube-api-access-rmktm" pod "79e292e4-cc4f-475f-8f95-8c3eb76fb5bc" (UID: "79e292e4-cc4f-475f-8f95-8c3eb76fb5bc"). InnerVolumeSpecName "kube-api-access-rmktm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:11:34.337829 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.337784    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:34.337829 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.337826    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmktm\" (UniqueName: \"kubernetes.io/projected/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc-kube-api-access-rmktm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:34.989804 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.989767    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k" event={"ID":"79e292e4-cc4f-475f-8f95-8c3eb76fb5bc","Type":"ContainerDied","Data":"bc15c655d18921456dd7da1fcb1ed3b18194f15dd5c8088aa12c5f89907860e8"}
Oct 31 20:11:34.990011 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.989822    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc15c655d18921456dd7da1fcb1ed3b18194f15dd5c8088aa12c5f89907860e8"
Oct 31 20:11:34.990011 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:34.989780    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"
Oct 31 20:11:35.395410 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:11:35.395366    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:11:38.601909 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:38.601872    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:43.579071 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:43.579031    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:47.812172 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:47.812130    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:49.451583 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:49.451536    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:49.667704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:49.667637    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:49.892833 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:49.892776    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:49.966966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:49.966935    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"]
Oct 31 20:11:50.027249 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:50.027209    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd95d7-66wqm"]
Oct 31 20:11:50.464700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:50.464636    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:51.597931 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:51.597898    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202" path="/var/lib/kubelet/pods/e5b94fe5-1fb8-4f08-b7bf-0e204ce2d202/volumes"
Oct 31 20:11:53.166476 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:53.166435    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:11:54.888248 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:54.888194    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:56.435843 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:56.435808    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:56.664499 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:56.664455    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:56.698492 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:56.698403    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"]
Oct 31 20:11:56.698681 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:56.698619    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy" containerID="cri-o://716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a" gracePeriod=30
Oct 31 20:11:57.609234 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.609200    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"]
Oct 31 20:11:57.609739 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.609722    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e7330358-d4c3-4f17-a61c-d794d4425a28" containerName="payload-pre-processing"
Oct 31 20:11:57.609802 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.609741    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7330358-d4c3-4f17-a61c-d794d4425a28" containerName="payload-pre-processing"
Oct 31 20:11:57.609802 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.609751    5006 cpu_manager.go:414] "containerMap: removing container" podUID="8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6" containerName="payload-processing"
Oct 31 20:11:57.609802 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.609757    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6" containerName="payload-processing"
Oct 31 20:11:57.609802 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.609766    5006 cpu_manager.go:414] "containerMap: removing container" podUID="79e292e4-cc4f-475f-8f95-8c3eb76fb5bc" containerName="revoke-keys"
Oct 31 20:11:57.609802 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.609770    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e292e4-cc4f-475f-8f95-8c3eb76fb5bc" containerName="revoke-keys"
Oct 31 20:11:57.609962 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.609875    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7330358-d4c3-4f17-a61c-d794d4425a28" containerName="payload-pre-processing"
Oct 31 20:11:57.609962 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.609882    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd31ef1-4e8e-4ba4-aaa5-ec5e7697f1e6" containerName="payload-processing"
Oct 31 20:11:57.609962 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.609888    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e292e4-cc4f-475f-8f95-8c3eb76fb5bc" containerName="revoke-keys"
Oct 31 20:11:57.613408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.613389    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.667080 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.667043    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"]
Oct 31 20:11:57.711878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.711838    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.712100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.711903    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjhqd\" (UniqueName: \"kubernetes.io/projected/11ae1713-feee-457f-96aa-afaf2085e886-kube-api-access-wjhqd\") pod \"maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.712100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.712022    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.714004 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.713979    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:11:57.714132 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.713986    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:11:57.714132 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.714090    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:11:57.721952 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.721927    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:11:57.812725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.812683    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.812910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.812817    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.812910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.812871    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjhqd\" (UniqueName: \"kubernetes.io/projected/11ae1713-feee-457f-96aa-afaf2085e886-kube-api-access-wjhqd\") pod \"maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.813429 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.813405    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.815118 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.815094    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.874988 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.874908    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjhqd\" (UniqueName: \"kubernetes.io/projected/11ae1713-feee-457f-96aa-afaf2085e886-kube-api-access-wjhqd\") pod \"maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.926903 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.926866    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:11:57.941783 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:57.941735    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:58.295860 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:11:58.295825    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ae1713_feee_457f_96aa_afaf2085e886.slice/crio-09f80c4c02886aa3f2b038321a3e0499385e9127ce8a82883f5e4b5e87f3b83e WatchSource:0}: Error finding container 09f80c4c02886aa3f2b038321a3e0499385e9127ce8a82883f5e4b5e87f3b83e: Status 404 returned error can't find the container with id 09f80c4c02886aa3f2b038321a3e0499385e9127ce8a82883f5e4b5e87f3b83e
Oct 31 20:11:58.298639 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:58.298614    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"]
Oct 31 20:11:59.104528 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.104488    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"]
Oct 31 20:11:59.149408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.149081    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w" event={"ID":"11ae1713-feee-457f-96aa-afaf2085e886","Type":"ContainerStarted","Data":"f91d1212878c7fb23de0a57efc3cea92a7765b9cb0be1539746c7c00b119a8cb"}
Oct 31 20:11:59.149408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.149156    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w" event={"ID":"11ae1713-feee-457f-96aa-afaf2085e886","Type":"ContainerStarted","Data":"09f80c4c02886aa3f2b038321a3e0499385e9127ce8a82883f5e4b5e87f3b83e"}
Oct 31 20:11:59.202645 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.202550    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:59.334373 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.334315    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w" podStartSLOduration=2.334302434 podStartE2EDuration="2.334302434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:11:59.333747504 +0000 UTC m=+3337.863281582" watchObservedRunningTime="2026-10-31 20:11:59.334302434 +0000 UTC m=+3337.863836543"
Oct 31 20:11:59.632315 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.632276    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"]
Oct 31 20:11:59.636320 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.636298    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:11:59.696425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.696388    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"]
Oct 31 20:11:59.732413 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.732377    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lw8p\" (UniqueName: \"kubernetes.io/projected/c343108d-802d-49f8-a9b4-c087f72a49c6-kube-api-access-7lw8p\") pod \"maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:11:59.732614 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.732432    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:11:59.732700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.732675    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:11:59.833864 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.833822    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lw8p\" (UniqueName: \"kubernetes.io/projected/c343108d-802d-49f8-a9b4-c087f72a49c6-kube-api-access-7lw8p\") pod \"maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:11:59.833864 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.833872    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:11:59.834101 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.833963    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:11:59.834589 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.834570    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:11:59.836433 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.836414    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:11:59.900162 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.900085    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lw8p\" (UniqueName: \"kubernetes.io/projected/c343108d-802d-49f8-a9b4-c087f72a49c6-kube-api-access-7lw8p\") pod \"maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:11:59.950037 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:11:59.950000    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:12:00.088499 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:00.088468    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:00.156486 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:00.156394    5006 generic.go:356] "Generic (PLEG): container finished" podID="11ae1713-feee-457f-96aa-afaf2085e886" containerID="f91d1212878c7fb23de0a57efc3cea92a7765b9cb0be1539746c7c00b119a8cb" exitCode=0
Oct 31 20:12:00.156913 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:00.156475    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w" event={"ID":"11ae1713-feee-457f-96aa-afaf2085e886","Type":"ContainerDied","Data":"f91d1212878c7fb23de0a57efc3cea92a7765b9cb0be1539746c7c00b119a8cb"}
Oct 31 20:12:00.323801 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:12:00.323760    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc343108d_802d_49f8_a9b4_c087f72a49c6.slice/crio-e0e29baf1994410319ce9266dccff8e7d625a185027f35e8d831b34adcf420fd WatchSource:0}: Error finding container e0e29baf1994410319ce9266dccff8e7d625a185027f35e8d831b34adcf420fd: Status 404 returned error can't find the container with id e0e29baf1994410319ce9266dccff8e7d625a185027f35e8d831b34adcf420fd
Oct 31 20:12:00.326535 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:00.326506    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"]
Oct 31 20:12:01.164726 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.164690    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj" event={"ID":"c343108d-802d-49f8-a9b4-c087f72a49c6","Type":"ContainerStarted","Data":"ea5d8a9c38f2e320d27b54053cf0d884f395c2530b9bc33a77689d33720c67ab"}
Oct 31 20:12:01.165227 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.164744    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj" event={"ID":"c343108d-802d-49f8-a9b4-c087f72a49c6","Type":"ContainerStarted","Data":"e0e29baf1994410319ce9266dccff8e7d625a185027f35e8d831b34adcf420fd"}
Oct 31 20:12:01.279989 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.279932    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj" podStartSLOduration=2.279914866 podStartE2EDuration="2.279914866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:12:01.277876968 +0000 UTC m=+3339.807411020" watchObservedRunningTime="2026-10-31 20:12:01.279914866 +0000 UTC m=+3339.809448913"
Oct 31 20:12:01.316100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.316075    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:12:01.358856 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.358812    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-xks-ca\") pod \"11ae1713-feee-457f-96aa-afaf2085e886\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") "
Oct 31 20:12:01.359037 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.358900    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ae1713-feee-457f-96aa-afaf2085e886-kube-api-access-wjhqd\" (UniqueName: \"kubernetes.io/projected/11ae1713-feee-457f-96aa-afaf2085e886-kube-api-access-wjhqd\") pod \"11ae1713-feee-457f-96aa-afaf2085e886\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") "
Oct 31 20:12:01.359037 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.358932    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-openshift-ca\") pod \"11ae1713-feee-457f-96aa-afaf2085e886\" (UID: \"11ae1713-feee-457f-96aa-afaf2085e886\") "
Oct 31 20:12:01.359362 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.359338    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-openshift-ca" pod "11ae1713-feee-457f-96aa-afaf2085e886" (UID: "11ae1713-feee-457f-96aa-afaf2085e886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:01.361211 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.361182    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ae1713-feee-457f-96aa-afaf2085e886-kube-api-access-wjhqd" pod "11ae1713-feee-457f-96aa-afaf2085e886" (UID: "11ae1713-feee-457f-96aa-afaf2085e886"). InnerVolumeSpecName "kube-api-access-wjhqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:01.361313 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.361262    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-xks-ca" pod "11ae1713-feee-457f-96aa-afaf2085e886" (UID: "11ae1713-feee-457f-96aa-afaf2085e886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:12:01.460289 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.460198    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:01.460289 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.460233    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjhqd\" (UniqueName: \"kubernetes.io/projected/11ae1713-feee-457f-96aa-afaf2085e886-kube-api-access-wjhqd\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:01.460289 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.460243    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ae1713-feee-457f-96aa-afaf2085e886-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:01.982046 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:01.981974    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:12:02.067243 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067207    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-data\") pod \"db340a3a-dca2-42cd-84de-01422845fd59\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") "
Oct 31 20:12:02.067445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067290    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-socket\") pod \"db340a3a-dca2-42cd-84de-01422845fd59\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") "
Oct 31 20:12:02.067445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067317    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-envoy\") pod \"db340a3a-dca2-42cd-84de-01422845fd59\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") "
Oct 31 20:12:02.067445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067354    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-credential-socket\") pod \"db340a3a-dca2-42cd-84de-01422845fd59\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") "
Oct 31 20:12:02.067445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067389    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-certs\") pod \"db340a3a-dca2-42cd-84de-01422845fd59\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") "
Oct 31 20:12:02.067445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067434    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db340a3a-dca2-42cd-84de-01422845fd59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db340a3a-dca2-42cd-84de-01422845fd59-istiod-ca-cert\") pod \"db340a3a-dca2-42cd-84de-01422845fd59\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") "
Oct 31 20:12:02.067778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067467    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db340a3a-dca2-42cd-84de-01422845fd59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db340a3a-dca2-42cd-84de-01422845fd59-istio-podinfo\") pod \"db340a3a-dca2-42cd-84de-01422845fd59\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") "
Oct 31 20:12:02.067778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067498    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-istio-token\" (UniqueName: \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-istio-token\") pod \"db340a3a-dca2-42cd-84de-01422845fd59\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") "
Oct 31 20:12:02.067778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067587    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-kube-api-access-qttq2\" (UniqueName: \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-kube-api-access-qttq2\") pod \"db340a3a-dca2-42cd-84de-01422845fd59\" (UID: \"db340a3a-dca2-42cd-84de-01422845fd59\") "
Oct 31 20:12:02.067778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067596    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-data" pod "db340a3a-dca2-42cd-84de-01422845fd59" (UID: "db340a3a-dca2-42cd-84de-01422845fd59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:02.067778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.067638    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-socket" pod "db340a3a-dca2-42cd-84de-01422845fd59" (UID: "db340a3a-dca2-42cd-84de-01422845fd59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:02.068099 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.068013    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:02.068099 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.068032    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:02.068230 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.068126    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-certs" pod "db340a3a-dca2-42cd-84de-01422845fd59" (UID: "db340a3a-dca2-42cd-84de-01422845fd59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:02.068230 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.068178    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db340a3a-dca2-42cd-84de-01422845fd59-istiod-ca-cert" pod "db340a3a-dca2-42cd-84de-01422845fd59" (UID: "db340a3a-dca2-42cd-84de-01422845fd59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:02.068371 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.068336    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-credential-socket" pod "db340a3a-dca2-42cd-84de-01422845fd59" (UID: "db340a3a-dca2-42cd-84de-01422845fd59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:02.070507 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.070482    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-envoy" pod "db340a3a-dca2-42cd-84de-01422845fd59" (UID: "db340a3a-dca2-42cd-84de-01422845fd59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:02.070814 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.070791    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db340a3a-dca2-42cd-84de-01422845fd59-istio-podinfo" pod "db340a3a-dca2-42cd-84de-01422845fd59" (UID: "db340a3a-dca2-42cd-84de-01422845fd59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:12:02.070910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.070820    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-kube-api-access-qttq2" pod "db340a3a-dca2-42cd-84de-01422845fd59" (UID: "db340a3a-dca2-42cd-84de-01422845fd59"). InnerVolumeSpecName "kube-api-access-qttq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:02.070989 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.070915    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-istio-token" pod "db340a3a-dca2-42cd-84de-01422845fd59" (UID: "db340a3a-dca2-42cd-84de-01422845fd59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:02.168841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.168808    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:02.168841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.168838    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:02.168841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.168847    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db340a3a-dca2-42cd-84de-01422845fd59-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:02.169347 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.168856    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db340a3a-dca2-42cd-84de-01422845fd59-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:02.169347 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.168865    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db340a3a-dca2-42cd-84de-01422845fd59-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:02.169347 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.168874    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:02.169347 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.168885    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qttq2\" (UniqueName: \"kubernetes.io/projected/db340a3a-dca2-42cd-84de-01422845fd59-kube-api-access-qttq2\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:02.172833 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.172802    5006 generic.go:356] "Generic (PLEG): container finished" podID="c343108d-802d-49f8-a9b4-c087f72a49c6" containerID="ea5d8a9c38f2e320d27b54053cf0d884f395c2530b9bc33a77689d33720c67ab" exitCode=0
Oct 31 20:12:02.172976 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.172887    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj" event={"ID":"c343108d-802d-49f8-a9b4-c087f72a49c6","Type":"ContainerDied","Data":"ea5d8a9c38f2e320d27b54053cf0d884f395c2530b9bc33a77689d33720c67ab"}
Oct 31 20:12:02.174441 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.174410    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"
Oct 31 20:12:02.174549 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.174458    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w" event={"ID":"11ae1713-feee-457f-96aa-afaf2085e886","Type":"ContainerDied","Data":"09f80c4c02886aa3f2b038321a3e0499385e9127ce8a82883f5e4b5e87f3b83e"}
Oct 31 20:12:02.174549 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.174508    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09f80c4c02886aa3f2b038321a3e0499385e9127ce8a82883f5e4b5e87f3b83e"
Oct 31 20:12:02.178400 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.178374    5006 generic.go:356] "Generic (PLEG): container finished" podID="db340a3a-dca2-42cd-84de-01422845fd59" containerID="716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a" exitCode=0
Oct 31 20:12:02.178510 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.178420    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"
Oct 31 20:12:02.178510 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.178461    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" event={"ID":"db340a3a-dca2-42cd-84de-01422845fd59","Type":"ContainerDied","Data":"716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a"}
Oct 31 20:12:02.178634 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.178518    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd" event={"ID":"db340a3a-dca2-42cd-84de-01422845fd59","Type":"ContainerDied","Data":"9c78c3d7269c969904795a5842d944bd1ff17e012fa4b9d88d26bfe99ba68e63"}
Oct 31 20:12:02.178634 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.178547    5006 scope.go:122] "RemoveContainer" containerID="716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a"
Oct 31 20:12:02.189520 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.189496    5006 scope.go:122] "RemoveContainer" containerID="522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc"
Oct 31 20:12:02.198764 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.198744    5006 scope.go:122] "RemoveContainer" containerID="716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a"
Oct 31 20:12:02.199092 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:12:02.199069    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a\": container with ID starting with 716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a not found: ID does not exist" containerID="716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a"
Oct 31 20:12:02.199161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.199104    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a"} err="failed to get container status \"716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a\": rpc error: code = NotFound desc = could not find container \"716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a\": container with ID starting with 716fcd00455b5032f8a1fa2553d8282253f88008a636e2320d162586e2543f5a not found: ID does not exist"
Oct 31 20:12:02.199161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.199128    5006 scope.go:122] "RemoveContainer" containerID="522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc"
Oct 31 20:12:02.199379 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:12:02.199365    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc\": container with ID starting with 522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc not found: ID does not exist" containerID="522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc"
Oct 31 20:12:02.199420 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.199383    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc"} err="failed to get container status \"522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc\": rpc error: code = NotFound desc = could not find container \"522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc\": container with ID starting with 522bcf9c6b9b91687ddfac502c79a3d8d49e00d1390c4b1da4a3759f8da5e8fc not found: ID does not exist"
Oct 31 20:12:02.369101 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.369067    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"]
Oct 31 20:12:02.372761 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:02.372733    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-n7czd"]
Oct 31 20:12:03.338336 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.338307    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:12:03.381345 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.381304    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-openshift-ca\") pod \"c343108d-802d-49f8-a9b4-c087f72a49c6\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") "
Oct 31 20:12:03.381566 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.381361    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-xks-ca\") pod \"c343108d-802d-49f8-a9b4-c087f72a49c6\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") "
Oct 31 20:12:03.381566 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.381391    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c343108d-802d-49f8-a9b4-c087f72a49c6-kube-api-access-7lw8p\" (UniqueName: \"kubernetes.io/projected/c343108d-802d-49f8-a9b4-c087f72a49c6-kube-api-access-7lw8p\") pod \"c343108d-802d-49f8-a9b4-c087f72a49c6\" (UID: \"c343108d-802d-49f8-a9b4-c087f72a49c6\") "
Oct 31 20:12:03.381747 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.381721    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-openshift-ca" pod "c343108d-802d-49f8-a9b4-c087f72a49c6" (UID: "c343108d-802d-49f8-a9b4-c087f72a49c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:03.381914 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.381898    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:03.383609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.383583    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-xks-ca" pod "c343108d-802d-49f8-a9b4-c087f72a49c6" (UID: "c343108d-802d-49f8-a9b4-c087f72a49c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:12:03.383609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.383594    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c343108d-802d-49f8-a9b4-c087f72a49c6-kube-api-access-7lw8p" pod "c343108d-802d-49f8-a9b4-c087f72a49c6" (UID: "c343108d-802d-49f8-a9b4-c087f72a49c6"). InnerVolumeSpecName "kube-api-access-7lw8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:03.483278 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.483185    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c343108d-802d-49f8-a9b4-c087f72a49c6-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:03.483278 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.483235    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lw8p\" (UniqueName: \"kubernetes.io/projected/c343108d-802d-49f8-a9b4-c087f72a49c6-kube-api-access-7lw8p\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:03.597789 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:03.597752    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db340a3a-dca2-42cd-84de-01422845fd59" path="/var/lib/kubelet/pods/db340a3a-dca2-42cd-84de-01422845fd59/volumes"
Oct 31 20:12:04.192519 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:04.192486    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:12:04.192699 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:04.192523    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj" event={"ID":"c343108d-802d-49f8-a9b4-c087f72a49c6","Type":"ContainerDied","Data":"e0e29baf1994410319ce9266dccff8e7d625a185027f35e8d831b34adcf420fd"}
Oct 31 20:12:04.192699 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:04.192570    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e29baf1994410319ce9266dccff8e7d625a185027f35e8d831b34adcf420fd"
Oct 31 20:12:05.115470 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:05.115433    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"]
Oct 31 20:12:05.165950 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:05.165911    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"]
Oct 31 20:12:05.168465 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:05.168440    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0930954e-z5t8w"]
Oct 31 20:12:05.170855 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:05.170830    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c674f1a-vdfkh"]
Oct 31 20:12:05.398980 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:12:05.398887    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:12:05.407776 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:05.407734    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:05.598465 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:05.598430    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cf565b-5277-48cb-bc3e-aeb9246df687" path="/var/lib/kubelet/pods/72cf565b-5277-48cb-bc3e-aeb9246df687/volumes"
Oct 31 20:12:05.598905 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:05.598891    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69ba68a-797e-4014-aa92-bd60d877492d" path="/var/lib/kubelet/pods/b69ba68a-797e-4014-aa92-bd60d877492d/volumes"
Oct 31 20:12:07.242144 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:07.242107    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:12:09.096638 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:09.096595    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"]
Oct 31 20:12:09.202946 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:09.202901    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"]
Oct 31 20:12:09.534490 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:09.534400    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:09.751180 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:09.751143    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:10.370873 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:10.370838    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:11.538308 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:11.538263    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"]
Oct 31 20:12:11.645365 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:11.645332    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"]
Oct 31 20:12:11.985995 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:11.985964    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:12.211286 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:12.211251    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:14.097754 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.097704    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6" podUID="f5e1347c-23bb-4951-aa24-0117b25e05df" containerName="payload-processing" containerID="cri-o://095726f5d506d5c2007d205950900eee1f64ed22626837b58d2e2d5504833fa6" gracePeriod=25
Oct 31 20:12:14.126081 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.126012    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz" podUID="23dc8445-ca59-4827-a21d-c11fe709e600" containerName="main" containerID="cri-o://8a44d92374a7f93d5702f49e73cfc77d0948418e41a149607c42dd3663a9abec" gracePeriod=45
Oct 31 20:12:14.203326 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.203251    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k" podUID="044047a1-281a-40e7-954b-86dc240804a3" containerName="payload-pre-processing" containerID="cri-o://4e0067051329dbf55a820cd4b596427d755faccb5d95f3a28f2fc3d7c0d463a8" gracePeriod=25
Oct 31 20:12:14.268775 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.268736    5006 generic.go:356] "Generic (PLEG): container finished" podID="f5e1347c-23bb-4951-aa24-0117b25e05df" containerID="095726f5d506d5c2007d205950900eee1f64ed22626837b58d2e2d5504833fa6" exitCode=0
Oct 31 20:12:14.269059 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.268966    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6" event={"ID":"f5e1347c-23bb-4951-aa24-0117b25e05df","Type":"ContainerDied","Data":"095726f5d506d5c2007d205950900eee1f64ed22626837b58d2e2d5504833fa6"}
Oct 31 20:12:14.270847 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.270815    5006 generic.go:356] "Generic (PLEG): container finished" podID="23dc8445-ca59-4827-a21d-c11fe709e600" containerID="8a44d92374a7f93d5702f49e73cfc77d0948418e41a149607c42dd3663a9abec" exitCode=0
Oct 31 20:12:14.271065 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.270877    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz" event={"ID":"23dc8445-ca59-4827-a21d-c11fe709e600","Type":"ContainerDied","Data":"8a44d92374a7f93d5702f49e73cfc77d0948418e41a149607c42dd3663a9abec"}
Oct 31 20:12:14.410668 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.410622    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:12:14.433402 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.433368    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:12:14.513936 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.513899    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23dc8445-ca59-4827-a21d-c11fe709e600-kube-api-access-nrwmg\" (UniqueName: \"kubernetes.io/projected/23dc8445-ca59-4827-a21d-c11fe709e600-kube-api-access-nrwmg\") pod \"23dc8445-ca59-4827-a21d-c11fe709e600\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") "
Oct 31 20:12:14.513936 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.513944    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-home\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-home\") pod \"23dc8445-ca59-4827-a21d-c11fe709e600\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") "
Oct 31 20:12:14.514228 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.513972    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e1347c-23bb-4951-aa24-0117b25e05df-kube-api-access-8k2wk\" (UniqueName: \"kubernetes.io/projected/f5e1347c-23bb-4951-aa24-0117b25e05df-kube-api-access-8k2wk\") pod \"f5e1347c-23bb-4951-aa24-0117b25e05df\" (UID: \"f5e1347c-23bb-4951-aa24-0117b25e05df\") "
Oct 31 20:12:14.514228 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.514157    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-dshm\") pod \"23dc8445-ca59-4827-a21d-c11fe709e600\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") "
Oct 31 20:12:14.514349 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.514271    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e1347c-23bb-4951-aa24-0117b25e05df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e1347c-23bb-4951-aa24-0117b25e05df-plugins-config-volume\") pod \"f5e1347c-23bb-4951-aa24-0117b25e05df\" (UID: \"f5e1347c-23bb-4951-aa24-0117b25e05df\") "
Oct 31 20:12:14.514349 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.514285    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-home" pod "23dc8445-ca59-4827-a21d-c11fe709e600" (UID: "23dc8445-ca59-4827-a21d-c11fe709e600"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:14.514349 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.514303    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-model-cache\") pod \"23dc8445-ca59-4827-a21d-c11fe709e600\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") "
Oct 31 20:12:14.514349 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.514337    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-tmp-dir\") pod \"23dc8445-ca59-4827-a21d-c11fe709e600\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") "
Oct 31 20:12:14.514568 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.514409    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23dc8445-ca59-4827-a21d-c11fe709e600-tls-certs\" (UniqueName: \"kubernetes.io/secret/23dc8445-ca59-4827-a21d-c11fe709e600-tls-certs\") pod \"23dc8445-ca59-4827-a21d-c11fe709e600\" (UID: \"23dc8445-ca59-4827-a21d-c11fe709e600\") "
Oct 31 20:12:14.514620 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.514594    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-model-cache" pod "23dc8445-ca59-4827-a21d-c11fe709e600" (UID: "23dc8445-ca59-4827-a21d-c11fe709e600"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:14.514692 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.514637    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e1347c-23bb-4951-aa24-0117b25e05df-plugins-config-volume" pod "f5e1347c-23bb-4951-aa24-0117b25e05df" (UID: "f5e1347c-23bb-4951-aa24-0117b25e05df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:14.514969 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.514905    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-tmp-dir" pod "23dc8445-ca59-4827-a21d-c11fe709e600" (UID: "23dc8445-ca59-4827-a21d-c11fe709e600"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:14.515120 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.515042    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:14.515120 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.515069    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e1347c-23bb-4951-aa24-0117b25e05df-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:14.515120 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.515088    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:14.515120 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.515104    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:14.516629 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.516599    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-dshm" pod "23dc8445-ca59-4827-a21d-c11fe709e600" (UID: "23dc8445-ca59-4827-a21d-c11fe709e600"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:14.516629 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.516617    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23dc8445-ca59-4827-a21d-c11fe709e600-tls-certs" pod "23dc8445-ca59-4827-a21d-c11fe709e600" (UID: "23dc8445-ca59-4827-a21d-c11fe709e600"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:12:14.516789 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.516721    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23dc8445-ca59-4827-a21d-c11fe709e600-kube-api-access-nrwmg" pod "23dc8445-ca59-4827-a21d-c11fe709e600" (UID: "23dc8445-ca59-4827-a21d-c11fe709e600"). InnerVolumeSpecName "kube-api-access-nrwmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:14.516832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.516799    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e1347c-23bb-4951-aa24-0117b25e05df-kube-api-access-8k2wk" pod "f5e1347c-23bb-4951-aa24-0117b25e05df" (UID: "f5e1347c-23bb-4951-aa24-0117b25e05df"). InnerVolumeSpecName "kube-api-access-8k2wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:14.616592 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.616496    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23dc8445-ca59-4827-a21d-c11fe709e600-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:14.616592 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.616535    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrwmg\" (UniqueName: \"kubernetes.io/projected/23dc8445-ca59-4827-a21d-c11fe709e600-kube-api-access-nrwmg\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:14.616592 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.616547    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k2wk\" (UniqueName: \"kubernetes.io/projected/f5e1347c-23bb-4951-aa24-0117b25e05df-kube-api-access-8k2wk\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:14.616592 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:14.616556    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23dc8445-ca59-4827-a21d-c11fe709e600-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:15.278629 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.278596    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"
Oct 31 20:12:15.279107 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.278637    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6" event={"ID":"f5e1347c-23bb-4951-aa24-0117b25e05df","Type":"ContainerDied","Data":"a1404b3811f48126412f5d0d80e19104ed773ac105e043b200e0b01c63a41a2b"}
Oct 31 20:12:15.279107 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.278745    5006 scope.go:122] "RemoveContainer" containerID="095726f5d506d5c2007d205950900eee1f64ed22626837b58d2e2d5504833fa6"
Oct 31 20:12:15.280589 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.280561    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz" event={"ID":"23dc8445-ca59-4827-a21d-c11fe709e600","Type":"ContainerDied","Data":"94283b77996914faa93fc656d34bb4c217934fe577637bdd31b43993122d9c08"}
Oct 31 20:12:15.280589 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.280581    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"
Oct 31 20:12:15.290530 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.290486    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:15.291273 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.291165    5006 scope.go:122] "RemoveContainer" containerID="8a44d92374a7f93d5702f49e73cfc77d0948418e41a149607c42dd3663a9abec"
Oct 31 20:12:15.447426 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.447328    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"]
Oct 31 20:12:15.449821 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.449799    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5d59a2-848bb8644c-zd7n6"]
Oct 31 20:12:15.598128 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.598094    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e1347c-23bb-4951-aa24-0117b25e05df" path="/var/lib/kubelet/pods/f5e1347c-23bb-4951-aa24-0117b25e05df/volumes"
Oct 31 20:12:15.607250 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.607217    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"]
Oct 31 20:12:15.609663 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:15.609623    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ef44dd/test-model-ef44dd-kserve-bf8884dfc-vrkrz"]
Oct 31 20:12:16.090393 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:16.090355    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k" podUID="044047a1-281a-40e7-954b-86dc240804a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:12:16.539543 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:16.539443    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl" podUID="e4404670-ccda-42bd-a961-fe617cfd4216" containerName="payload-processing" containerID="cri-o://463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7" gracePeriod=25
Oct 31 20:12:16.646539 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:16.646489    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww" podUID="27f871dc-fee7-45b8-922a-ae1cbf0491ce" containerName="payload-pre-processing" containerID="cri-o://e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2" gracePeriod=25
Oct 31 20:12:16.810219 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:16.810184    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:12:16.939935 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:16.939898    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4404670-ccda-42bd-a961-fe617cfd4216-kube-api-access-sj4xt\" (UniqueName: \"kubernetes.io/projected/e4404670-ccda-42bd-a961-fe617cfd4216-kube-api-access-sj4xt\") pod \"e4404670-ccda-42bd-a961-fe617cfd4216\" (UID: \"e4404670-ccda-42bd-a961-fe617cfd4216\") "
Oct 31 20:12:16.940154 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:16.939976    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4404670-ccda-42bd-a961-fe617cfd4216-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4404670-ccda-42bd-a961-fe617cfd4216-plugins-config-volume\") pod \"e4404670-ccda-42bd-a961-fe617cfd4216\" (UID: \"e4404670-ccda-42bd-a961-fe617cfd4216\") "
Oct 31 20:12:16.940423 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:16.940399    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4404670-ccda-42bd-a961-fe617cfd4216-plugins-config-volume" pod "e4404670-ccda-42bd-a961-fe617cfd4216" (UID: "e4404670-ccda-42bd-a961-fe617cfd4216"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:16.942313 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:16.942283    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4404670-ccda-42bd-a961-fe617cfd4216-kube-api-access-sj4xt" pod "e4404670-ccda-42bd-a961-fe617cfd4216" (UID: "e4404670-ccda-42bd-a961-fe617cfd4216"). InnerVolumeSpecName "kube-api-access-sj4xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:17.041445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.041345    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj4xt\" (UniqueName: \"kubernetes.io/projected/e4404670-ccda-42bd-a961-fe617cfd4216-kube-api-access-sj4xt\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:17.041445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.041381    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4404670-ccda-42bd-a961-fe617cfd4216-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:17.296645 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.296554    5006 generic.go:356] "Generic (PLEG): container finished" podID="e4404670-ccda-42bd-a961-fe617cfd4216" containerID="463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7" exitCode=0
Oct 31 20:12:17.296645 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.296602    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"
Oct 31 20:12:17.296858 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.296639    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl" event={"ID":"e4404670-ccda-42bd-a961-fe617cfd4216","Type":"ContainerDied","Data":"463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7"}
Oct 31 20:12:17.296858 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.296708    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl" event={"ID":"e4404670-ccda-42bd-a961-fe617cfd4216","Type":"ContainerDied","Data":"c28ab309d0f14e131c7a85abc317cf8bd12a41e90e6c4cb5768ff7ee1390459d"}
Oct 31 20:12:17.296858 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.296730    5006 scope.go:122] "RemoveContainer" containerID="463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7"
Oct 31 20:12:17.309462 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.309427    5006 scope.go:122] "RemoveContainer" containerID="463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7"
Oct 31 20:12:17.309905 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:12:17.309876    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7\": container with ID starting with 463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7 not found: ID does not exist" containerID="463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7"
Oct 31 20:12:17.310019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.309915    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7"} err="failed to get container status \"463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7\": rpc error: code = NotFound desc = could not find container \"463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7\": container with ID starting with 463c716f571af09c0584256b8fa9e2b96934b4dcd3331b9a5b0dec5bc4d546f7 not found: ID does not exist"
Oct 31 20:12:17.462709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.462668    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"]
Oct 31 20:12:17.512940 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.512900    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a89a58-5ddbbc6bf-q5gbl"]
Oct 31 20:12:17.598262 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.598228    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23dc8445-ca59-4827-a21d-c11fe709e600" path="/var/lib/kubelet/pods/23dc8445-ca59-4827-a21d-c11fe709e600/volumes"
Oct 31 20:12:17.598672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:17.598638    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4404670-ccda-42bd-a961-fe617cfd4216" path="/var/lib/kubelet/pods/e4404670-ccda-42bd-a961-fe617cfd4216/volumes"
Oct 31 20:12:19.548375 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:19.548321    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:21.121412 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:21.121379    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:21.386641 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:21.386554    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:21.557125 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:21.557089    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"]
Oct 31 20:12:21.607332 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:21.607291    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"]
Oct 31 20:12:21.612618 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:21.612588    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:21.645954 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:12:21.645857    5006 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj\" is forbidden: User \"system:node:ip-10-0-128-104.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-104.us-west-2.compute.internal' and this object" podUID="c343108d-802d-49f8-a9b4-c087f72a49c6" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a89a58-f82lj"
Oct 31 20:12:21.887302 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:21.887266    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww" podUID="27f871dc-fee7-45b8-922a-ae1cbf0491ce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:12:22.049102 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:22.049009    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:23.601888 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:23.601849    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c343108d-802d-49f8-a9b4-c087f72a49c6" path="/var/lib/kubelet/pods/c343108d-802d-49f8-a9b4-c087f72a49c6/volumes"
Oct 31 20:12:24.727266 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:24.727225    5006 scope.go:122] "RemoveContainer" containerID="352a65a634a4d27a28fa06b7bbaf40f4bfe1353943579f24babfa618e3a43565"
Oct 31 20:12:24.736992 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:24.736970    5006 scope.go:122] "RemoveContainer" containerID="4f35a1edc65de2f2e61198ef7cf8b9f51c4eb10079b870a701f09fd05da1d85a"
Oct 31 20:12:24.746473 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:24.746447    5006 scope.go:122] "RemoveContainer" containerID="d29baf2059486f2c53a58ad54766d561579c5ce6fafc4e6f983bd9c63cbcd1ca"
Oct 31 20:12:24.756004 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:24.755978    5006 scope.go:122] "RemoveContainer" containerID="05fdce719222c840b3fbfa5deb37c88714c98b37bc02ea053b4202920659ddd0"
Oct 31 20:12:26.090525 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:26.090480    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k" podUID="044047a1-281a-40e7-954b-86dc240804a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:12:26.579022 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:26.578937    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:27.188729 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:27.188644    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:27.423365 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:27.423330    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:28.025020 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:28.024975    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:28.249370 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:28.249339    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:28.266035 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:28.265994    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"]
Oct 31 20:12:28.266230 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:28.266206    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy" containerID="cri-o://e7e030e0f3d1871ba5a50de6f4bcfa7fb5fd6f075c061fa85d2deea4b8740330" gracePeriod=30
Oct 31 20:12:28.462868 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:28.462832    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:29.115511 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.115470    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"]
Oct 31 20:12:29.115982 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.115968    5006 cpu_manager.go:414] "containerMap: removing container" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy"
Oct 31 20:12:29.116040 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.115983    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy"
Oct 31 20:12:29.116040 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.115998    5006 cpu_manager.go:414] "containerMap: removing container" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy"
Oct 31 20:12:29.116040 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116002    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy"
Oct 31 20:12:29.116040 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116015    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e4404670-ccda-42bd-a961-fe617cfd4216" containerName="payload-processing"
Oct 31 20:12:29.116040 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116021    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4404670-ccda-42bd-a961-fe617cfd4216" containerName="payload-processing"
Oct 31 20:12:29.116040 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116038    5006 cpu_manager.go:414] "containerMap: removing container" podUID="f5e1347c-23bb-4951-aa24-0117b25e05df" containerName="payload-processing"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116043    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e1347c-23bb-4951-aa24-0117b25e05df" containerName="payload-processing"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116050    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c343108d-802d-49f8-a9b4-c087f72a49c6" containerName="revoke-keys"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116055    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c343108d-802d-49f8-a9b4-c087f72a49c6" containerName="revoke-keys"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116062    5006 cpu_manager.go:414] "containerMap: removing container" podUID="11ae1713-feee-457f-96aa-afaf2085e886" containerName="revoke-keys"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116066    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ae1713-feee-457f-96aa-afaf2085e886" containerName="revoke-keys"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116074    5006 cpu_manager.go:414] "containerMap: removing container" podUID="23dc8445-ca59-4827-a21d-c11fe709e600" containerName="main"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116080    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23dc8445-ca59-4827-a21d-c11fe709e600" containerName="main"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116150    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116158    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="db340a3a-dca2-42cd-84de-01422845fd59" containerName="istio-proxy"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116166    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4404670-ccda-42bd-a961-fe617cfd4216" containerName="payload-processing"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116175    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e1347c-23bb-4951-aa24-0117b25e05df" containerName="payload-processing"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116180    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c343108d-802d-49f8-a9b4-c087f72a49c6" containerName="revoke-keys"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116186    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ae1713-feee-457f-96aa-afaf2085e886" containerName="revoke-keys"
Oct 31 20:12:29.116238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.116192    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="23dc8445-ca59-4827-a21d-c11fe709e600" containerName="main"
Oct 31 20:12:29.121297 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.121279    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.173491 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.173460    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"]
Oct 31 20:12:29.221410 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.221367    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:12:29.221611 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.221444    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:12:29.243550 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.243511    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:12:29.271867 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.271830    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:12:29.275330 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.275300    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de16c014-6735-4a15-8436-5517f27c8869-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.275469 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.275351    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de16c014-6735-4a15-8436-5517f27c8869-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.275562 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.275535    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6gn\" (UniqueName: \"kubernetes.io/projected/de16c014-6735-4a15-8436-5517f27c8869-kube-api-access-td6gn\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.376563 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.376476    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de16c014-6735-4a15-8436-5517f27c8869-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.376563 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.376516    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de16c014-6735-4a15-8436-5517f27c8869-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.376778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.376579    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6gn\" (UniqueName: \"kubernetes.io/projected/de16c014-6735-4a15-8436-5517f27c8869-kube-api-access-td6gn\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.377330 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.377308    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de16c014-6735-4a15-8436-5517f27c8869-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.379081 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.379056    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de16c014-6735-4a15-8436-5517f27c8869-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.434974 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.434935    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6gn\" (UniqueName: \"kubernetes.io/projected/de16c014-6735-4a15-8436-5517f27c8869-kube-api-access-td6gn\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.435145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.435128    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:29.812334 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:12:29.812302    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde16c014_6735_4a15_8436_5517f27c8869.slice/crio-dcaa2e31b27ecd21a821195135b0c15713347fe5ac879c6b434c2117867a8b7d WatchSource:0}: Error finding container dcaa2e31b27ecd21a821195135b0c15713347fe5ac879c6b434c2117867a8b7d: Status 404 returned error can't find the container with id dcaa2e31b27ecd21a821195135b0c15713347fe5ac879c6b434c2117867a8b7d
Oct 31 20:12:29.813129 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:29.813088    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"]
Oct 31 20:12:30.163693 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:30.163638    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:12:30.385600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:30.385503    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw" event={"ID":"de16c014-6735-4a15-8436-5517f27c8869","Type":"ContainerStarted","Data":"59f182507180c76c2b9b45c3d7da8ddfd5403cccf6d36ea09dd4151fb5cebb5a"}
Oct 31 20:12:30.385600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:30.385576    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw" event={"ID":"de16c014-6735-4a15-8436-5517f27c8869","Type":"ContainerStarted","Data":"dcaa2e31b27ecd21a821195135b0c15713347fe5ac879c6b434c2117867a8b7d"}
Oct 31 20:12:30.500451 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:30.500393    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw" podStartSLOduration=1.500377769 podStartE2EDuration="1.500377769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:12:30.499255491 +0000 UTC m=+3369.028789526" watchObservedRunningTime="2026-10-31 20:12:30.500377769 +0000 UTC m=+3369.029911882"
Oct 31 20:12:30.674291 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:30.674259    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:31.116598 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:31.116561    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:31.396373 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:31.396277    5006 generic.go:356] "Generic (PLEG): container finished" podID="de16c014-6735-4a15-8436-5517f27c8869" containerID="59f182507180c76c2b9b45c3d7da8ddfd5403cccf6d36ea09dd4151fb5cebb5a" exitCode=0
Oct 31 20:12:31.396799 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:31.396364    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw" event={"ID":"de16c014-6735-4a15-8436-5517f27c8869","Type":"ContainerDied","Data":"59f182507180c76c2b9b45c3d7da8ddfd5403cccf6d36ea09dd4151fb5cebb5a"}
Oct 31 20:12:31.887050 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:31.887002    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww" podUID="27f871dc-fee7-45b8-922a-ae1cbf0491ce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:12:32.563455 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.563421    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:32.607046 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.607005    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de16c014-6735-4a15-8436-5517f27c8869-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de16c014-6735-4a15-8436-5517f27c8869-maas-api-xks-ca\") pod \"de16c014-6735-4a15-8436-5517f27c8869\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") "
Oct 31 20:12:32.607237 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.607082    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de16c014-6735-4a15-8436-5517f27c8869-kube-api-access-td6gn\" (UniqueName: \"kubernetes.io/projected/de16c014-6735-4a15-8436-5517f27c8869-kube-api-access-td6gn\") pod \"de16c014-6735-4a15-8436-5517f27c8869\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") "
Oct 31 20:12:32.607237 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.607170    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de16c014-6735-4a15-8436-5517f27c8869-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de16c014-6735-4a15-8436-5517f27c8869-maas-api-openshift-ca\") pod \"de16c014-6735-4a15-8436-5517f27c8869\" (UID: \"de16c014-6735-4a15-8436-5517f27c8869\") "
Oct 31 20:12:32.607549 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.607521    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de16c014-6735-4a15-8436-5517f27c8869-maas-api-openshift-ca" pod "de16c014-6735-4a15-8436-5517f27c8869" (UID: "de16c014-6735-4a15-8436-5517f27c8869"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:32.609283 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.609244    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de16c014-6735-4a15-8436-5517f27c8869-kube-api-access-td6gn" pod "de16c014-6735-4a15-8436-5517f27c8869" (UID: "de16c014-6735-4a15-8436-5517f27c8869"). InnerVolumeSpecName "kube-api-access-td6gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:32.609407 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.609293    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de16c014-6735-4a15-8436-5517f27c8869-maas-api-xks-ca" pod "de16c014-6735-4a15-8436-5517f27c8869" (UID: "de16c014-6735-4a15-8436-5517f27c8869"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:12:32.709048 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.708942    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de16c014-6735-4a15-8436-5517f27c8869-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:32.709048 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.708980    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de16c014-6735-4a15-8436-5517f27c8869-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:32.709048 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.708991    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td6gn\" (UniqueName: \"kubernetes.io/projected/de16c014-6735-4a15-8436-5517f27c8869-kube-api-access-td6gn\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:32.835475 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:32.835435    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:33.071886 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.071803    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:33.302181 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.302148    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:33.347960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.347923    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"]
Oct 31 20:12:33.350494 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.350465    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d59a2-zt78w"]
Oct 31 20:12:33.414000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.413959    5006 generic.go:356] "Generic (PLEG): container finished" podID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerID="e7e030e0f3d1871ba5a50de6f4bcfa7fb5fd6f075c061fa85d2deea4b8740330" exitCode=0
Oct 31 20:12:33.414182 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.414030    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" event={"ID":"24d8c84b-aefa-4e42-b121-33efa295c4b2","Type":"ContainerDied","Data":"e7e030e0f3d1871ba5a50de6f4bcfa7fb5fd6f075c061fa85d2deea4b8740330"}
Oct 31 20:12:33.414182 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.414089    5006 scope.go:122] "RemoveContainer" containerID="f84eea95a62cbd4f5bfed203f94c14fe9baaef7fe72aa61f743c8a28b2224982"
Oct 31 20:12:33.415880 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.415845    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw" event={"ID":"de16c014-6735-4a15-8436-5517f27c8869","Type":"ContainerDied","Data":"dcaa2e31b27ecd21a821195135b0c15713347fe5ac879c6b434c2117867a8b7d"}
Oct 31 20:12:33.416024 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.415903    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"
Oct 31 20:12:33.416024 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.415906    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcaa2e31b27ecd21a821195135b0c15713347fe5ac879c6b434c2117867a8b7d"
Oct 31 20:12:33.529511 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.529480    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:12:33.598439 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.598355    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ae1713-feee-457f-96aa-afaf2085e886" path="/var/lib/kubelet/pods/11ae1713-feee-457f-96aa-afaf2085e886/volumes"
Oct 31 20:12:33.618929 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.618878    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-kube-api-access-4l8ft\" (UniqueName: \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-kube-api-access-4l8ft\") pod \"24d8c84b-aefa-4e42-b121-33efa295c4b2\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") "
Oct 31 20:12:33.619125 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.618975    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-certs\") pod \"24d8c84b-aefa-4e42-b121-33efa295c4b2\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") "
Oct 31 20:12:33.619125 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.618998    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-credential-socket\") pod \"24d8c84b-aefa-4e42-b121-33efa295c4b2\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") "
Oct 31 20:12:33.619125 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619062    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-token\" (UniqueName: \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-token\") pod \"24d8c84b-aefa-4e42-b121-33efa295c4b2\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") "
Oct 31 20:12:33.619125 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619079    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-podinfo\") pod \"24d8c84b-aefa-4e42-b121-33efa295c4b2\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") "
Oct 31 20:12:33.619125 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619113    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-socket\") pod \"24d8c84b-aefa-4e42-b121-33efa295c4b2\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") "
Oct 31 20:12:33.619411 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619140    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d8c84b-aefa-4e42-b121-33efa295c4b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d8c84b-aefa-4e42-b121-33efa295c4b2-istiod-ca-cert\") pod \"24d8c84b-aefa-4e42-b121-33efa295c4b2\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") "
Oct 31 20:12:33.619411 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619188    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-data\") pod \"24d8c84b-aefa-4e42-b121-33efa295c4b2\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") "
Oct 31 20:12:33.619411 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619222    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-envoy\") pod \"24d8c84b-aefa-4e42-b121-33efa295c4b2\" (UID: \"24d8c84b-aefa-4e42-b121-33efa295c4b2\") "
Oct 31 20:12:33.619411 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619274    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-certs" pod "24d8c84b-aefa-4e42-b121-33efa295c4b2" (UID: "24d8c84b-aefa-4e42-b121-33efa295c4b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:33.619620 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619421    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-credential-socket" pod "24d8c84b-aefa-4e42-b121-33efa295c4b2" (UID: "24d8c84b-aefa-4e42-b121-33efa295c4b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:33.619620 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619518    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-data" pod "24d8c84b-aefa-4e42-b121-33efa295c4b2" (UID: "24d8c84b-aefa-4e42-b121-33efa295c4b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:33.619758 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619636    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d8c84b-aefa-4e42-b121-33efa295c4b2-istiod-ca-cert" pod "24d8c84b-aefa-4e42-b121-33efa295c4b2" (UID: "24d8c84b-aefa-4e42-b121-33efa295c4b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:33.619758 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619680    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:33.619758 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619701    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:33.619758 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619717    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:33.619928 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.619844    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-socket" pod "24d8c84b-aefa-4e42-b121-33efa295c4b2" (UID: "24d8c84b-aefa-4e42-b121-33efa295c4b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:33.621358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.621329    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-podinfo" pod "24d8c84b-aefa-4e42-b121-33efa295c4b2" (UID: "24d8c84b-aefa-4e42-b121-33efa295c4b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:12:33.621463 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.621398    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-token" pod "24d8c84b-aefa-4e42-b121-33efa295c4b2" (UID: "24d8c84b-aefa-4e42-b121-33efa295c4b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:33.621753 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.621736    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-envoy" pod "24d8c84b-aefa-4e42-b121-33efa295c4b2" (UID: "24d8c84b-aefa-4e42-b121-33efa295c4b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:12:33.621806 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.621770    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-kube-api-access-4l8ft" pod "24d8c84b-aefa-4e42-b121-33efa295c4b2" (UID: "24d8c84b-aefa-4e42-b121-33efa295c4b2"). InnerVolumeSpecName "kube-api-access-4l8ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:33.720771 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.720723    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l8ft\" (UniqueName: \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-kube-api-access-4l8ft\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:33.720771 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.720762    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:33.720771 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.720776    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:33.721040 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.720788    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:33.721040 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.720801    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d8c84b-aefa-4e42-b121-33efa295c4b2-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:33.721040 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.720814    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d8c84b-aefa-4e42-b121-33efa295c4b2-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:33.793338 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:33.793295    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:34.425176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.425139    5006 generic.go:356] "Generic (PLEG): container finished" podID="044047a1-281a-40e7-954b-86dc240804a3" containerID="4e0067051329dbf55a820cd4b596427d755faccb5d95f3a28f2fc3d7c0d463a8" exitCode=0
Oct 31 20:12:34.425393 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.425220    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k" event={"ID":"044047a1-281a-40e7-954b-86dc240804a3","Type":"ContainerDied","Data":"4e0067051329dbf55a820cd4b596427d755faccb5d95f3a28f2fc3d7c0d463a8"}
Oct 31 20:12:34.427175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.427141    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr" event={"ID":"24d8c84b-aefa-4e42-b121-33efa295c4b2","Type":"ContainerDied","Data":"59e7dce5a2e06e5821ba37dfefe6ccc80c3ec6be21bfe5b319fd03046e451643"}
Oct 31 20:12:34.427345 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.427212    5006 scope.go:122] "RemoveContainer" containerID="e7e030e0f3d1871ba5a50de6f4bcfa7fb5fd6f075c061fa85d2deea4b8740330"
Oct 31 20:12:34.427345 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.427152    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"
Oct 31 20:12:34.470851 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.470771    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:12:34.528504 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.528460    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044047a1-281a-40e7-954b-86dc240804a3-kube-api-access-vhrhj\" (UniqueName: \"kubernetes.io/projected/044047a1-281a-40e7-954b-86dc240804a3-kube-api-access-vhrhj\") pod \"044047a1-281a-40e7-954b-86dc240804a3\" (UID: \"044047a1-281a-40e7-954b-86dc240804a3\") "
Oct 31 20:12:34.528769 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.528529    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044047a1-281a-40e7-954b-86dc240804a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044047a1-281a-40e7-954b-86dc240804a3-plugins-config-volume\") pod \"044047a1-281a-40e7-954b-86dc240804a3\" (UID: \"044047a1-281a-40e7-954b-86dc240804a3\") "
Oct 31 20:12:34.528936 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.528912    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044047a1-281a-40e7-954b-86dc240804a3-plugins-config-volume" pod "044047a1-281a-40e7-954b-86dc240804a3" (UID: "044047a1-281a-40e7-954b-86dc240804a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:34.530774 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.530740    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044047a1-281a-40e7-954b-86dc240804a3-kube-api-access-vhrhj" pod "044047a1-281a-40e7-954b-86dc240804a3" (UID: "044047a1-281a-40e7-954b-86dc240804a3"). InnerVolumeSpecName "kube-api-access-vhrhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:34.630000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.629917    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhrhj\" (UniqueName: \"kubernetes.io/projected/044047a1-281a-40e7-954b-86dc240804a3-kube-api-access-vhrhj\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:34.630000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.629956    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044047a1-281a-40e7-954b-86dc240804a3-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:34.761766 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.761680    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"]
Oct 31 20:12:34.765700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:34.765667    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a89a58-openshift-default-68964c4d4c-xw4rr"]
Oct 31 20:12:35.394716 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:12:35.394673    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:12:35.436533 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:35.436493    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k" event={"ID":"044047a1-281a-40e7-954b-86dc240804a3","Type":"ContainerDied","Data":"72da15fb09f3e28591f67c9273b42abc02369b87149140b0859bbec786af3403"}
Oct 31 20:12:35.436533 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:35.436532    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"
Oct 31 20:12:35.436842 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:35.436587    5006 scope.go:122] "RemoveContainer" containerID="4e0067051329dbf55a820cd4b596427d755faccb5d95f3a28f2fc3d7c0d463a8"
Oct 31 20:12:35.599282 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:35.599245    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" path="/var/lib/kubelet/pods/24d8c84b-aefa-4e42-b121-33efa295c4b2/volumes"
Oct 31 20:12:35.600898 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:35.600870    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"]
Oct 31 20:12:35.651288 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:35.651211    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d59a2-d87548f7f-rkz8k"]
Oct 31 20:12:36.909885 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:36.909815    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:12:36.956057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:36.956020    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f871dc-fee7-45b8-922a-ae1cbf0491ce-kube-api-access-pbghp\" (UniqueName: \"kubernetes.io/projected/27f871dc-fee7-45b8-922a-ae1cbf0491ce-kube-api-access-pbghp\") pod \"27f871dc-fee7-45b8-922a-ae1cbf0491ce\" (UID: \"27f871dc-fee7-45b8-922a-ae1cbf0491ce\") "
Oct 31 20:12:36.956228 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:36.956169    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f871dc-fee7-45b8-922a-ae1cbf0491ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f871dc-fee7-45b8-922a-ae1cbf0491ce-plugins-config-volume\") pod \"27f871dc-fee7-45b8-922a-ae1cbf0491ce\" (UID: \"27f871dc-fee7-45b8-922a-ae1cbf0491ce\") "
Oct 31 20:12:36.956561 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:36.956528    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f871dc-fee7-45b8-922a-ae1cbf0491ce-plugins-config-volume" pod "27f871dc-fee7-45b8-922a-ae1cbf0491ce" (UID: "27f871dc-fee7-45b8-922a-ae1cbf0491ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:36.958308 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:36.958273    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f871dc-fee7-45b8-922a-ae1cbf0491ce-kube-api-access-pbghp" pod "27f871dc-fee7-45b8-922a-ae1cbf0491ce" (UID: "27f871dc-fee7-45b8-922a-ae1cbf0491ce"). InnerVolumeSpecName "kube-api-access-pbghp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:37.057849 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.057795    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbghp\" (UniqueName: \"kubernetes.io/projected/27f871dc-fee7-45b8-922a-ae1cbf0491ce-kube-api-access-pbghp\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:37.057849 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.057841    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f871dc-fee7-45b8-922a-ae1cbf0491ce-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:37.450864 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.450824    5006 generic.go:356] "Generic (PLEG): container finished" podID="27f871dc-fee7-45b8-922a-ae1cbf0491ce" containerID="e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2" exitCode=0
Oct 31 20:12:37.451084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.450877    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"
Oct 31 20:12:37.451084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.450912    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww" event={"ID":"27f871dc-fee7-45b8-922a-ae1cbf0491ce","Type":"ContainerDied","Data":"e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2"}
Oct 31 20:12:37.451084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.450990    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww" event={"ID":"27f871dc-fee7-45b8-922a-ae1cbf0491ce","Type":"ContainerDied","Data":"38e53bb862411d42c090d6a78d8a8e279646bce53d9c904acd95d5892d1dae1d"}
Oct 31 20:12:37.451084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.451021    5006 scope.go:122] "RemoveContainer" containerID="e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2"
Oct 31 20:12:37.461137 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.461117    5006 scope.go:122] "RemoveContainer" containerID="e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2"
Oct 31 20:12:37.461431 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:12:37.461410    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2\": container with ID starting with e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2 not found: ID does not exist" containerID="e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2"
Oct 31 20:12:37.461505 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.461439    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2"} err="failed to get container status \"e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2\": rpc error: code = NotFound desc = could not find container \"e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2\": container with ID starting with e9c7973caf7ee0f052a720678dc6658089a6d0078e69ad649d460fc9867ceeb2 not found: ID does not exist"
Oct 31 20:12:37.597545 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.597504    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044047a1-281a-40e7-954b-86dc240804a3" path="/var/lib/kubelet/pods/044047a1-281a-40e7-954b-86dc240804a3/volumes"
Oct 31 20:12:37.616699 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.616628    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"]
Oct 31 20:12:37.666960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:37.666923    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a89a58-b668944b7-8jlww"]
Oct 31 20:12:38.414988 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:38.414937    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:39.597863 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:39.597826    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f871dc-fee7-45b8-922a-ae1cbf0491ce" path="/var/lib/kubelet/pods/27f871dc-fee7-45b8-922a-ae1cbf0491ce/volumes"
Oct 31 20:12:39.699142 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:39.699093    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:40.015701 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:40.015593    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:40.244598 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:40.244563    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:40.578544 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:40.578504    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:41.005736 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.005698    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"]
Oct 31 20:12:41.006951 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.006918    5006 cpu_manager.go:414] "containerMap: removing container" podUID="044047a1-281a-40e7-954b-86dc240804a3" containerName="payload-pre-processing"
Oct 31 20:12:41.006951 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.006950    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044047a1-281a-40e7-954b-86dc240804a3" containerName="payload-pre-processing"
Oct 31 20:12:41.007176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.006977    5006 cpu_manager.go:414] "containerMap: removing container" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy"
Oct 31 20:12:41.007176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.006986    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy"
Oct 31 20:12:41.007176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007014    5006 cpu_manager.go:414] "containerMap: removing container" podUID="27f871dc-fee7-45b8-922a-ae1cbf0491ce" containerName="payload-pre-processing"
Oct 31 20:12:41.007176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007022    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f871dc-fee7-45b8-922a-ae1cbf0491ce" containerName="payload-pre-processing"
Oct 31 20:12:41.007176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007063    5006 cpu_manager.go:414] "containerMap: removing container" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy"
Oct 31 20:12:41.007176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007071    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy"
Oct 31 20:12:41.007176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007087    5006 cpu_manager.go:414] "containerMap: removing container" podUID="de16c014-6735-4a15-8436-5517f27c8869" containerName="revoke-keys"
Oct 31 20:12:41.007176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007095    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de16c014-6735-4a15-8436-5517f27c8869" containerName="revoke-keys"
Oct 31 20:12:41.007670 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007206    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f871dc-fee7-45b8-922a-ae1cbf0491ce" containerName="payload-pre-processing"
Oct 31 20:12:41.007670 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007246    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy"
Oct 31 20:12:41.007670 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007258    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="de16c014-6735-4a15-8436-5517f27c8869" containerName="revoke-keys"
Oct 31 20:12:41.007670 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007269    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="044047a1-281a-40e7-954b-86dc240804a3" containerName="payload-pre-processing"
Oct 31 20:12:41.007670 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.007279    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d8c84b-aefa-4e42-b121-33efa295c4b2" containerName="istio-proxy"
Oct 31 20:12:41.013165 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.013137    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.060886 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.060850    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"]
Oct 31 20:12:41.098055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.098012    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.098055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.098062    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rghv\" (UniqueName: \"kubernetes.io/projected/e852e734-c7b6-42a2-93b6-f7276bcedf63-kube-api-access-2rghv\") pod \"maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.098340 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.098192    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.108437 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.108393    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:12:41.108663 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.108535    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:12:41.119924 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.119891    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:12:41.119924 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.119903    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:12:41.199157 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.199113    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.199157 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.199177    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rghv\" (UniqueName: \"kubernetes.io/projected/e852e734-c7b6-42a2-93b6-f7276bcedf63-kube-api-access-2rghv\") pod \"maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.199563 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.199246    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.200134 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.200107    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.201815 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.201790    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.256741 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.256665    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rghv\" (UniqueName: \"kubernetes.io/projected/e852e734-c7b6-42a2-93b6-f7276bcedf63-kube-api-access-2rghv\") pod \"maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.344679 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.344615    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:41.714568 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:12:41.714531    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode852e734_c7b6_42a2_93b6_f7276bcedf63.slice/crio-fa0b3244f19777a0b6b8979b82cc54b2163429a4f4942b02c7a07d4f08735a89 WatchSource:0}: Error finding container fa0b3244f19777a0b6b8979b82cc54b2163429a4f4942b02c7a07d4f08735a89: Status 404 returned error can't find the container with id fa0b3244f19777a0b6b8979b82cc54b2163429a4f4942b02c7a07d4f08735a89
Oct 31 20:12:41.716983 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:41.716961    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"]
Oct 31 20:12:42.482887 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:42.482837    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6" event={"ID":"e852e734-c7b6-42a2-93b6-f7276bcedf63","Type":"ContainerStarted","Data":"56ad4b29a1ec345be886450828b71df5c3f89e4e846166210dac050826117f48"}
Oct 31 20:12:42.483311 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:42.482908    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6" event={"ID":"e852e734-c7b6-42a2-93b6-f7276bcedf63","Type":"ContainerStarted","Data":"fa0b3244f19777a0b6b8979b82cc54b2163429a4f4942b02c7a07d4f08735a89"}
Oct 31 20:12:42.597550 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:42.597194    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6" podStartSLOduration=2.597177678 podStartE2EDuration="2.597177678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:12:42.596241491 +0000 UTC m=+3381.125775544" watchObservedRunningTime="2026-10-31 20:12:42.597177678 +0000 UTC m=+3381.126711732"
Oct 31 20:12:43.490234 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:43.490200    5006 generic.go:356] "Generic (PLEG): container finished" podID="e852e734-c7b6-42a2-93b6-f7276bcedf63" containerID="56ad4b29a1ec345be886450828b71df5c3f89e4e846166210dac050826117f48" exitCode=0
Oct 31 20:12:43.490677 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:43.490288    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6" event={"ID":"e852e734-c7b6-42a2-93b6-f7276bcedf63","Type":"ContainerDied","Data":"56ad4b29a1ec345be886450828b71df5c3f89e4e846166210dac050826117f48"}
Oct 31 20:12:43.711192 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:43.711158    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:44.642497 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:44.642466    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:44.737920 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:44.737886    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-xks-ca\") pod \"e852e734-c7b6-42a2-93b6-f7276bcedf63\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") "
Oct 31 20:12:44.738108 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:44.737934    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-openshift-ca\") pod \"e852e734-c7b6-42a2-93b6-f7276bcedf63\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") "
Oct 31 20:12:44.738108 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:44.737986    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e852e734-c7b6-42a2-93b6-f7276bcedf63-kube-api-access-2rghv\" (UniqueName: \"kubernetes.io/projected/e852e734-c7b6-42a2-93b6-f7276bcedf63-kube-api-access-2rghv\") pod \"e852e734-c7b6-42a2-93b6-f7276bcedf63\" (UID: \"e852e734-c7b6-42a2-93b6-f7276bcedf63\") "
Oct 31 20:12:44.738415 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:44.738384    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-openshift-ca" pod "e852e734-c7b6-42a2-93b6-f7276bcedf63" (UID: "e852e734-c7b6-42a2-93b6-f7276bcedf63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:44.740121 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:44.740100    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-xks-ca" pod "e852e734-c7b6-42a2-93b6-f7276bcedf63" (UID: "e852e734-c7b6-42a2-93b6-f7276bcedf63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:12:44.740306 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:44.740289    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e852e734-c7b6-42a2-93b6-f7276bcedf63-kube-api-access-2rghv" pod "e852e734-c7b6-42a2-93b6-f7276bcedf63" (UID: "e852e734-c7b6-42a2-93b6-f7276bcedf63"). InnerVolumeSpecName "kube-api-access-2rghv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:44.838868 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:44.838832    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:44.838868 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:44.838863    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e852e734-c7b6-42a2-93b6-f7276bcedf63-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:44.838868 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:44.838874    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rghv\" (UniqueName: \"kubernetes.io/projected/e852e734-c7b6-42a2-93b6-f7276bcedf63-kube-api-access-2rghv\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:45.415886 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:45.415851    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"]
Oct 31 20:12:45.503029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:45.502992    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6" event={"ID":"e852e734-c7b6-42a2-93b6-f7276bcedf63","Type":"ContainerDied","Data":"fa0b3244f19777a0b6b8979b82cc54b2163429a4f4942b02c7a07d4f08735a89"}
Oct 31 20:12:45.503182 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:45.503044    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"
Oct 31 20:12:45.503182 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:45.503051    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa0b3244f19777a0b6b8979b82cc54b2163429a4f4942b02c7a07d4f08735a89"
Oct 31 20:12:45.606739 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:45.606665    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:49.077926 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:49.077889    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:50.134421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:50.134378    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:12:52.602842 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:52.602800    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"]
Oct 31 20:12:52.949185 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:52.949091    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:53.166759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:53.166721    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:54.091796 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:54.091761    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:56.049852 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:56.049815    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"]
Oct 31 20:12:56.099834 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:56.099798    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b208fcd6-gnx9h"]
Oct 31 20:12:57.598075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:57.598039    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b369441-a0e9-4337-b7ac-cde5ba300eec" path="/var/lib/kubelet/pods/1b369441-a0e9-4337-b7ac-cde5ba300eec/volumes"
Oct 31 20:12:57.603926 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:57.603892    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk" podUID="41e920c5-f1ee-4aaa-ae85-88fa1e4aea58" containerName="payload-pre-processing" containerID="cri-o://99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387" gracePeriod=25
Oct 31 20:12:58.446775 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:12:58.446733    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:00.073744 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.073699    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:00.324459 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.324358    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:00.436717 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.436620    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main" containerID="cri-o://50638469c3ff06f0fad4736738d7641635bde54cdcb906afb9977e80cce99715" gracePeriod=45
Oct 31 20:13:00.463563 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.463526    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"]
Oct 31 20:13:00.513709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.513672    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44dd-27pqw"]
Oct 31 20:13:00.559548 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.559513    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:00.605041 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.605010    5006 generic.go:356] "Generic (PLEG): container finished" podID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerID="50638469c3ff06f0fad4736738d7641635bde54cdcb906afb9977e80cce99715" exitCode=0
Oct 31 20:13:00.605237 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.605098    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" event={"ID":"9498e552-f1ed-4372-8da2-07058bbeaf58","Type":"ContainerDied","Data":"50638469c3ff06f0fad4736738d7641635bde54cdcb906afb9977e80cce99715"}
Oct 31 20:13:00.715908 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.715878    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 20:13:00.814006 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.813969    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-tmp-dir\") pod \"9498e552-f1ed-4372-8da2-07058bbeaf58\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") "
Oct 31 20:13:00.814221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.814067    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-home\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-home\") pod \"9498e552-f1ed-4372-8da2-07058bbeaf58\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") "
Oct 31 20:13:00.814221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.814098    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-dshm\") pod \"9498e552-f1ed-4372-8da2-07058bbeaf58\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") "
Oct 31 20:13:00.814221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.814135    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9498e552-f1ed-4372-8da2-07058bbeaf58-kube-api-access-r2shz\" (UniqueName: \"kubernetes.io/projected/9498e552-f1ed-4372-8da2-07058bbeaf58-kube-api-access-r2shz\") pod \"9498e552-f1ed-4372-8da2-07058bbeaf58\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") "
Oct 31 20:13:00.814221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.814164    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9498e552-f1ed-4372-8da2-07058bbeaf58-tls-certs\" (UniqueName: \"kubernetes.io/secret/9498e552-f1ed-4372-8da2-07058bbeaf58-tls-certs\") pod \"9498e552-f1ed-4372-8da2-07058bbeaf58\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") "
Oct 31 20:13:00.814445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.814290    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-model-cache\") pod \"9498e552-f1ed-4372-8da2-07058bbeaf58\" (UID: \"9498e552-f1ed-4372-8da2-07058bbeaf58\") "
Oct 31 20:13:00.814445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.814327    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-tmp-dir" pod "9498e552-f1ed-4372-8da2-07058bbeaf58" (UID: "9498e552-f1ed-4372-8da2-07058bbeaf58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:13:00.814445 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.814379    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-home" pod "9498e552-f1ed-4372-8da2-07058bbeaf58" (UID: "9498e552-f1ed-4372-8da2-07058bbeaf58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:13:00.814642 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.814615    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:00.814737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.814640    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:00.814737 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.814676    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-model-cache" pod "9498e552-f1ed-4372-8da2-07058bbeaf58" (UID: "9498e552-f1ed-4372-8da2-07058bbeaf58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:13:00.816572 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.816531    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9498e552-f1ed-4372-8da2-07058bbeaf58-tls-certs" pod "9498e552-f1ed-4372-8da2-07058bbeaf58" (UID: "9498e552-f1ed-4372-8da2-07058bbeaf58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:13:00.816719 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.816571    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-dshm" pod "9498e552-f1ed-4372-8da2-07058bbeaf58" (UID: "9498e552-f1ed-4372-8da2-07058bbeaf58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:13:00.816719 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.816684    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9498e552-f1ed-4372-8da2-07058bbeaf58-kube-api-access-r2shz" pod "9498e552-f1ed-4372-8da2-07058bbeaf58" (UID: "9498e552-f1ed-4372-8da2-07058bbeaf58"). InnerVolumeSpecName "kube-api-access-r2shz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:13:00.915458 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.915378    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:00.915458 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.915412    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2shz\" (UniqueName: \"kubernetes.io/projected/9498e552-f1ed-4372-8da2-07058bbeaf58-kube-api-access-r2shz\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:00.915458 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.915422    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9498e552-f1ed-4372-8da2-07058bbeaf58-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:00.915458 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.915433    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9498e552-f1ed-4372-8da2-07058bbeaf58-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:00.945990 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:00.945958    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:01.027844 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:01.027796    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk" podUID="41e920c5-f1ee-4aaa-ae85-88fa1e4aea58" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:13:01.598510 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:01.598473    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de16c014-6735-4a15-8436-5517f27c8869" path="/var/lib/kubelet/pods/de16c014-6735-4a15-8436-5517f27c8869/volumes"
Oct 31 20:13:01.607693 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:13:01.607639    5006 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9498e552_f1ed_4372_8da2_07058bbeaf58.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9498e552_f1ed_4372_8da2_07058bbeaf58.slice/crio-f40a5acf85e042e01328f7b1a575d4b43369fc1ff3417b46b54d4103b75dc669\": RecentStats: unable to find data in memory cache]"
Oct 31 20:13:01.612876 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:01.612848    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl" event={"ID":"9498e552-f1ed-4372-8da2-07058bbeaf58","Type":"ContainerDied","Data":"f40a5acf85e042e01328f7b1a575d4b43369fc1ff3417b46b54d4103b75dc669"}
Oct 31 20:13:01.612876 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:01.612873    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"
Oct 31 20:13:01.613084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:01.612890    5006 scope.go:122] "RemoveContainer" containerID="50638469c3ff06f0fad4736738d7641635bde54cdcb906afb9977e80cce99715"
Oct 31 20:13:01.825801 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:01.825765    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"]
Oct 31 20:13:01.828409 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:01.828379    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-2-w2-967dee-kserve-7c6bdccb5d-zz8nl"]
Oct 31 20:13:03.597775 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:03.597735    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" path="/var/lib/kubelet/pods/9498e552-f1ed-4372-8da2-07058bbeaf58/volumes"
Oct 31 20:13:04.756021 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:04.753770    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:13:05.394720 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:13:05.394681    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:13:05.566170 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:05.566134    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:06.676252 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:06.676215    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:06.893176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:06.893142    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:07.132559 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.132480    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:07.786683 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.786636    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"]
Oct 31 20:13:07.787108 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.787095    5006 cpu_manager.go:414] "containerMap: removing container" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main"
Oct 31 20:13:07.787158 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.787109    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main"
Oct 31 20:13:07.787158 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.787130    5006 cpu_manager.go:414] "containerMap: removing container" podUID="e852e734-c7b6-42a2-93b6-f7276bcedf63" containerName="revoke-keys"
Oct 31 20:13:07.787158 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.787135    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e852e734-c7b6-42a2-93b6-f7276bcedf63" containerName="revoke-keys"
Oct 31 20:13:07.787259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.787211    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9498e552-f1ed-4372-8da2-07058bbeaf58" containerName="main"
Oct 31 20:13:07.787259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.787223    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e852e734-c7b6-42a2-93b6-f7276bcedf63" containerName="revoke-keys"
Oct 31 20:13:07.792274 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.792250    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:07.846553 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.846519    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"]
Oct 31 20:13:07.889131 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.889089    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:07.889131 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.889134    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:07.889384 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.889165    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgngh\" (UniqueName: \"kubernetes.io/projected/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-kube-api-access-zgngh\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:07.894549 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.894514    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:13:07.894619 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.894546    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:13:07.894619 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.894555    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:13:07.895256 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.895236    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:13:07.990075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.990027    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgngh\" (UniqueName: \"kubernetes.io/projected/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-kube-api-access-zgngh\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:07.990300 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.990155    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:07.990300 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.990177    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:07.990960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.990933    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:07.992705 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:07.992682    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:08.047571 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:08.047491    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgngh\" (UniqueName: \"kubernetes.io/projected/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-kube-api-access-zgngh\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:08.106383 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:08.106337    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:08.472261 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:13:08.472231    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2314fd_ffc6_45cf_9ffe_7226d81f278c.slice/crio-f3ee1fdc07e86a7fe1f7430f8e0ec2342065734516f32b676ae35188c07dfedc WatchSource:0}: Error finding container f3ee1fdc07e86a7fe1f7430f8e0ec2342065734516f32b676ae35188c07dfedc: Status 404 returned error can't find the container with id f3ee1fdc07e86a7fe1f7430f8e0ec2342065734516f32b676ae35188c07dfedc
Oct 31 20:13:08.474750 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:08.474725    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"]
Oct 31 20:13:08.658348 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:08.658305    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm" event={"ID":"2f2314fd-ffc6-45cf-9ffe-7226d81f278c","Type":"ContainerStarted","Data":"f3ee1fdc07e86a7fe1f7430f8e0ec2342065734516f32b676ae35188c07dfedc"}
Oct 31 20:13:09.665358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:09.665323    5006 generic.go:356] "Generic (PLEG): container finished" podID="2f2314fd-ffc6-45cf-9ffe-7226d81f278c" containerID="0e594dd278cdcaec536fde43af1a8fbbf2aba9c2df4359d854bf4889142f956d" exitCode=0
Oct 31 20:13:09.665800 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:09.665408    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm" event={"ID":"2f2314fd-ffc6-45cf-9ffe-7226d81f278c","Type":"ContainerDied","Data":"0e594dd278cdcaec536fde43af1a8fbbf2aba9c2df4359d854bf4889142f956d"}
Oct 31 20:13:10.811274 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:10.811246    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:10.879126 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:10.879088    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:10.922049 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:10.921955    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-xks-ca\") pod \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") "
Oct 31 20:13:10.922049 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:10.922023    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-kube-api-access-zgngh\" (UniqueName: \"kubernetes.io/projected/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-kube-api-access-zgngh\") pod \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") "
Oct 31 20:13:10.922309 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:10.922080    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-openshift-ca\") pod \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\" (UID: \"2f2314fd-ffc6-45cf-9ffe-7226d81f278c\") "
Oct 31 20:13:10.922627 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:10.922604    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-openshift-ca" pod "2f2314fd-ffc6-45cf-9ffe-7226d81f278c" (UID: "2f2314fd-ffc6-45cf-9ffe-7226d81f278c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:13:10.925204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:10.925146    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-kube-api-access-zgngh" pod "2f2314fd-ffc6-45cf-9ffe-7226d81f278c" (UID: "2f2314fd-ffc6-45cf-9ffe-7226d81f278c"). InnerVolumeSpecName "kube-api-access-zgngh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:13:10.925204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:10.925148    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-xks-ca" pod "2f2314fd-ffc6-45cf-9ffe-7226d81f278c" (UID: "2f2314fd-ffc6-45cf-9ffe-7226d81f278c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:13:11.023736 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:11.023687    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:11.023736 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:11.023727    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgngh\" (UniqueName: \"kubernetes.io/projected/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-kube-api-access-zgngh\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:11.023736 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:11.023739    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2314fd-ffc6-45cf-9ffe-7226d81f278c-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:11.028507 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:11.028480    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk" podUID="41e920c5-f1ee-4aaa-ae85-88fa1e4aea58" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:13:11.678164 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:11.678134    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"
Oct 31 20:13:11.678325 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:11.678135    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm" event={"ID":"2f2314fd-ffc6-45cf-9ffe-7226d81f278c","Type":"ContainerDied","Data":"f3ee1fdc07e86a7fe1f7430f8e0ec2342065734516f32b676ae35188c07dfedc"}
Oct 31 20:13:11.678325 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:11.678249    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3ee1fdc07e86a7fe1f7430f8e0ec2342065734516f32b676ae35188c07dfedc"
Oct 31 20:13:15.507845 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.507811    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"]
Oct 31 20:13:15.508244 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.508235    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2f2314fd-ffc6-45cf-9ffe-7226d81f278c" containerName="revoke-keys"
Oct 31 20:13:15.508296 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.508246    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2314fd-ffc6-45cf-9ffe-7226d81f278c" containerName="revoke-keys"
Oct 31 20:13:15.508355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.508345    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2314fd-ffc6-45cf-9ffe-7226d81f278c" containerName="revoke-keys"
Oct 31 20:13:15.511854 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.511830    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.568740 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.567883    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"]
Oct 31 20:13:15.568740 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.568456    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09cb56cf-5tkk7\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.568740 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.568537    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftcfx\" (UniqueName: \"kubernetes.io/projected/c751b999-02bd-4f36-814e-da70d5a2d94a-kube-api-access-ftcfx\") pod \"maas-api-revoke-sub-09cb56cf-5tkk7\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.568740 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.568576    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09cb56cf-5tkk7\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.614434 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.614388    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:13:15.614434 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.614396    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:13:15.614434 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.614425    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:13:15.616875 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.616853    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"]
Oct 31 20:13:15.617108 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.617094    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:13:15.621192 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.620781    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:15.631695 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.627633    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"]
Oct 31 20:13:15.669259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.669222    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftcfx\" (UniqueName: \"kubernetes.io/projected/c751b999-02bd-4f36-814e-da70d5a2d94a-kube-api-access-ftcfx\") pod \"maas-api-revoke-sub-09cb56cf-5tkk7\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.669259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.669262    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09cb56cf-5tkk7\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.669527 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.669287    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf56t\" (UniqueName: \"kubernetes.io/projected/75efbf1e-64e0-481d-b41c-1941975c5ff1-kube-api-access-nf56t\") pod \"maas-api-revoke-sub-5799008e-pw4cn\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:15.669527 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.669387    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5799008e-pw4cn\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:15.669527 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.669514    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09cb56cf-5tkk7\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.669822 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.669554    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5799008e-pw4cn\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:15.670175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.670156    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09cb56cf-5tkk7\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.671772 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.671749    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09cb56cf-5tkk7\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.728274 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.728234    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftcfx\" (UniqueName: \"kubernetes.io/projected/c751b999-02bd-4f36-814e-da70d5a2d94a-kube-api-access-ftcfx\") pod \"maas-api-revoke-sub-09cb56cf-5tkk7\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.771291 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.771191    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5799008e-pw4cn\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:15.771291 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.771270    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf56t\" (UniqueName: \"kubernetes.io/projected/75efbf1e-64e0-481d-b41c-1941975c5ff1-kube-api-access-nf56t\") pod \"maas-api-revoke-sub-5799008e-pw4cn\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:15.771558 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.771330    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5799008e-pw4cn\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:15.771931 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.771898    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5799008e-pw4cn\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:15.773717 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.773699    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5799008e-pw4cn\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:15.826878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.826838    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:15.828711 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.828683    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf56t\" (UniqueName: \"kubernetes.io/projected/75efbf1e-64e0-481d-b41c-1941975c5ff1-kube-api-access-nf56t\") pod \"maas-api-revoke-sub-5799008e-pw4cn\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:15.938109 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:15.938060    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:16.208536 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:13:16.208489    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc751b999_02bd_4f36_814e_da70d5a2d94a.slice/crio-4d6a29a2890b56165bec7cbf23bc3c9ee01223cf4548328ed83bc9af4b7b0c02 WatchSource:0}: Error finding container 4d6a29a2890b56165bec7cbf23bc3c9ee01223cf4548328ed83bc9af4b7b0c02: Status 404 returned error can't find the container with id 4d6a29a2890b56165bec7cbf23bc3c9ee01223cf4548328ed83bc9af4b7b0c02
Oct 31 20:13:16.210304 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:16.210265    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"]
Oct 31 20:13:16.316550 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:13:16.316501    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75efbf1e_64e0_481d_b41c_1941975c5ff1.slice/crio-6413957cf161a872cc8582b622eba331d9aaca79ca4648b0900d8179ad0a8778 WatchSource:0}: Error finding container 6413957cf161a872cc8582b622eba331d9aaca79ca4648b0900d8179ad0a8778: Status 404 returned error can't find the container with id 6413957cf161a872cc8582b622eba331d9aaca79ca4648b0900d8179ad0a8778
Oct 31 20:13:16.318939 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:16.318913    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"]
Oct 31 20:13:16.550351 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:16.550319    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:16.709723 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:16.709693    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7" event={"ID":"c751b999-02bd-4f36-814e-da70d5a2d94a","Type":"ContainerStarted","Data":"a53996e1824f8f9c626744c0c6e0e148ee6451dbe66f5cf1a2158dfddd03d68e"}
Oct 31 20:13:16.709858 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:16.709750    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7" event={"ID":"c751b999-02bd-4f36-814e-da70d5a2d94a","Type":"ContainerStarted","Data":"4d6a29a2890b56165bec7cbf23bc3c9ee01223cf4548328ed83bc9af4b7b0c02"}
Oct 31 20:13:16.711491 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:16.711466    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn" event={"ID":"75efbf1e-64e0-481d-b41c-1941975c5ff1","Type":"ContainerStarted","Data":"6413957cf161a872cc8582b622eba331d9aaca79ca4648b0900d8179ad0a8778"}
Oct 31 20:13:16.823723 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:16.823586    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7" podStartSLOduration=1.823572682 podStartE2EDuration="1.823572682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:13:16.822979119 +0000 UTC m=+3415.352513192" watchObservedRunningTime="2026-10-31 20:13:16.823572682 +0000 UTC m=+3415.353106734"
Oct 31 20:13:17.719400 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:17.719368    5006 generic.go:356] "Generic (PLEG): container finished" podID="c751b999-02bd-4f36-814e-da70d5a2d94a" containerID="a53996e1824f8f9c626744c0c6e0e148ee6451dbe66f5cf1a2158dfddd03d68e" exitCode=0
Oct 31 20:13:17.719829 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:17.719421    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7" event={"ID":"c751b999-02bd-4f36-814e-da70d5a2d94a","Type":"ContainerDied","Data":"a53996e1824f8f9c626744c0c6e0e148ee6451dbe66f5cf1a2158dfddd03d68e"}
Oct 31 20:13:17.721197 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:17.721166    5006 generic.go:356] "Generic (PLEG): container finished" podID="75efbf1e-64e0-481d-b41c-1941975c5ff1" containerID="0cf0fe7b190c97d0644148e9cdff1a163d8518e68d1c30d91fe1df39a1b89aea" exitCode=0
Oct 31 20:13:17.721319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:17.721209    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn" event={"ID":"75efbf1e-64e0-481d-b41c-1941975c5ff1","Type":"ContainerDied","Data":"0cf0fe7b190c97d0644148e9cdff1a163d8518e68d1c30d91fe1df39a1b89aea"}
Oct 31 20:13:17.879717 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:17.879687    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 20:13:17.995390 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:17.995358    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-kube-api-access-kd68c\" (UniqueName: \"kubernetes.io/projected/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-kube-api-access-kd68c\") pod \"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58\" (UID: \"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58\") "
Oct 31 20:13:17.995555 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:17.995408    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-plugins-config-volume\") pod \"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58\" (UID: \"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58\") "
Oct 31 20:13:17.995910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:17.995877    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-plugins-config-volume" pod "41e920c5-f1ee-4aaa-ae85-88fa1e4aea58" (UID: "41e920c5-f1ee-4aaa-ae85-88fa1e4aea58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:13:17.997545 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:17.997522    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-kube-api-access-kd68c" pod "41e920c5-f1ee-4aaa-ae85-88fa1e4aea58" (UID: "41e920c5-f1ee-4aaa-ae85-88fa1e4aea58"). InnerVolumeSpecName "kube-api-access-kd68c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:13:18.096978 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.096940    5006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-plugins-config-volume\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:18.096978 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.096973    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd68c\" (UniqueName: \"kubernetes.io/projected/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58-kube-api-access-kd68c\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:18.728906 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.728874    5006 generic.go:356] "Generic (PLEG): container finished" podID="41e920c5-f1ee-4aaa-ae85-88fa1e4aea58" containerID="99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387" exitCode=0
Oct 31 20:13:18.729382 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.728918    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"
Oct 31 20:13:18.729382 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.728951    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk" event={"ID":"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58","Type":"ContainerDied","Data":"99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387"}
Oct 31 20:13:18.729382 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.729004    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk" event={"ID":"41e920c5-f1ee-4aaa-ae85-88fa1e4aea58","Type":"ContainerDied","Data":"40f896458fd627ae01598ec4910e2ec727b1e3f2412c91b2072b766b12b6ccab"}
Oct 31 20:13:18.729382 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.729027    5006 scope.go:122] "RemoveContainer" containerID="99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387"
Oct 31 20:13:18.740391 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.740366    5006 scope.go:122] "RemoveContainer" containerID="99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387"
Oct 31 20:13:18.740795 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:13:18.740770    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387\": container with ID starting with 99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387 not found: ID does not exist" containerID="99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387"
Oct 31 20:13:18.740883 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.740806    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387"} err="failed to get container status \"99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387\": rpc error: code = NotFound desc = could not find container \"99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387\": container with ID starting with 99d370cb403d98138d41b30a43dd38f44e5a83de4928e5e674f8e57ea38cd387 not found: ID does not exist"
Oct 31 20:13:18.894829 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.894797    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"]
Oct 31 20:13:18.902723 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.902696    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:18.905170 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.905147    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:18.944935 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:18.944900    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-967dee-7f564bf579-6lkmk"]
Oct 31 20:13:19.005480 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.005366    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-openshift-ca\") pod \"c751b999-02bd-4f36-814e-da70d5a2d94a\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") "
Oct 31 20:13:19.005480 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.005474    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-xks-ca\") pod \"75efbf1e-64e0-481d-b41c-1941975c5ff1\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") "
Oct 31 20:13:19.005703 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.005514    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c751b999-02bd-4f36-814e-da70d5a2d94a-kube-api-access-ftcfx\" (UniqueName: \"kubernetes.io/projected/c751b999-02bd-4f36-814e-da70d5a2d94a-kube-api-access-ftcfx\") pod \"c751b999-02bd-4f36-814e-da70d5a2d94a\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") "
Oct 31 20:13:19.005703 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.005562    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75efbf1e-64e0-481d-b41c-1941975c5ff1-kube-api-access-nf56t\" (UniqueName: \"kubernetes.io/projected/75efbf1e-64e0-481d-b41c-1941975c5ff1-kube-api-access-nf56t\") pod \"75efbf1e-64e0-481d-b41c-1941975c5ff1\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") "
Oct 31 20:13:19.005703 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.005608    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-xks-ca\") pod \"c751b999-02bd-4f36-814e-da70d5a2d94a\" (UID: \"c751b999-02bd-4f36-814e-da70d5a2d94a\") "
Oct 31 20:13:19.005898 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.005710    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-openshift-ca\") pod \"75efbf1e-64e0-481d-b41c-1941975c5ff1\" (UID: \"75efbf1e-64e0-481d-b41c-1941975c5ff1\") "
Oct 31 20:13:19.005898 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.005838    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-openshift-ca" pod "c751b999-02bd-4f36-814e-da70d5a2d94a" (UID: "c751b999-02bd-4f36-814e-da70d5a2d94a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:13:19.006101 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.006073    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-openshift-ca" pod "75efbf1e-64e0-481d-b41c-1941975c5ff1" (UID: "75efbf1e-64e0-481d-b41c-1941975c5ff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:13:19.006201 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.006181    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:19.006245 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.006208    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:19.007935 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.007909    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c751b999-02bd-4f36-814e-da70d5a2d94a-kube-api-access-ftcfx" pod "c751b999-02bd-4f36-814e-da70d5a2d94a" (UID: "c751b999-02bd-4f36-814e-da70d5a2d94a"). InnerVolumeSpecName "kube-api-access-ftcfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:13:19.007935 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.007925    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-xks-ca" pod "c751b999-02bd-4f36-814e-da70d5a2d94a" (UID: "c751b999-02bd-4f36-814e-da70d5a2d94a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:13:19.008140 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.008122    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-xks-ca" pod "75efbf1e-64e0-481d-b41c-1941975c5ff1" (UID: "75efbf1e-64e0-481d-b41c-1941975c5ff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:13:19.008188 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.008143    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75efbf1e-64e0-481d-b41c-1941975c5ff1-kube-api-access-nf56t" pod "75efbf1e-64e0-481d-b41c-1941975c5ff1" (UID: "75efbf1e-64e0-481d-b41c-1941975c5ff1"). InnerVolumeSpecName "kube-api-access-nf56t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:13:19.107735 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.107697    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75efbf1e-64e0-481d-b41c-1941975c5ff1-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:19.107735 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.107731    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftcfx\" (UniqueName: \"kubernetes.io/projected/c751b999-02bd-4f36-814e-da70d5a2d94a-kube-api-access-ftcfx\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:19.107735 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.107741    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf56t\" (UniqueName: \"kubernetes.io/projected/75efbf1e-64e0-481d-b41c-1941975c5ff1-kube-api-access-nf56t\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:19.107998 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.107752    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751b999-02bd-4f36-814e-da70d5a2d94a-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:19.599993 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.599950    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e920c5-f1ee-4aaa-ae85-88fa1e4aea58" path="/var/lib/kubelet/pods/41e920c5-f1ee-4aaa-ae85-88fa1e4aea58/volumes"
Oct 31 20:13:19.727377 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.727336    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:19.738613 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.738579    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"
Oct 31 20:13:19.738613 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.738605    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7" event={"ID":"c751b999-02bd-4f36-814e-da70d5a2d94a","Type":"ContainerDied","Data":"4d6a29a2890b56165bec7cbf23bc3c9ee01223cf4548328ed83bc9af4b7b0c02"}
Oct 31 20:13:19.739117 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.738665    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d6a29a2890b56165bec7cbf23bc3c9ee01223cf4548328ed83bc9af4b7b0c02"
Oct 31 20:13:19.739875 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.739850    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn" event={"ID":"75efbf1e-64e0-481d-b41c-1941975c5ff1","Type":"ContainerDied","Data":"6413957cf161a872cc8582b622eba331d9aaca79ca4648b0900d8179ad0a8778"}
Oct 31 20:13:19.740034 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.739893    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6413957cf161a872cc8582b622eba331d9aaca79ca4648b0900d8179ad0a8778"
Oct 31 20:13:19.740034 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.739913    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"
Oct 31 20:13:19.947514 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:19.947426    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:21.577105 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:21.577068    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:22.282354 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:22.282316    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:22.507220 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:22.507183    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:23.386979 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:23.386943    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:24.828110 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:24.828074    5006 scope.go:122] "RemoveContainer" containerID="325c8f835f51cc1d71c895565cdde865e9170eb41a46330260d2e94c59b5c36b"
Oct 31 20:13:24.837816 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:24.837786    5006 scope.go:122] "RemoveContainer" containerID="d693b5ea1bc86b5bd58bb941e547acab1d99c0f3413560513b61705643ec21fc"
Oct 31 20:13:24.847854 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:24.847822    5006 scope.go:122] "RemoveContainer" containerID="7ddeab1bb2575f93e597bc2b81fbdb1fdf96e3442faaa264ff67ca6d158274bb"
Oct 31 20:13:25.985419 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:25.985382    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:27.620875 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:27.620808    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:27.854861 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:27.854822    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:27.977194 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:27.977097    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"]
Oct 31 20:13:28.027380 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:28.027340    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-872fee-srzkm"]
Oct 31 20:13:28.443003 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:28.442970    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:29.375464 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:29.375430    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:29.596634 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:29.596596    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:29.598360 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:29.598332    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2314fd-ffc6-45cf-9ffe-7226d81f278c" path="/var/lib/kubelet/pods/2f2314fd-ffc6-45cf-9ffe-7226d81f278c/volumes"
Oct 31 20:13:31.008177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:31.008134    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:31.221554 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:31.221517    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:13:32.374918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:32.374877    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:33.852543 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:33.852511    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:34.108561 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:34.108471    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:34.347641 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:34.347607    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"]
Oct 31 20:13:34.360324 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:34.360241    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:34.398294 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:34.398250    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-967dee-78cb6"]
Oct 31 20:13:34.815939 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:34.815828    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:35.082579 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.082475    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:35.387061 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.386984    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:35.394081 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:13:35.394048    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:13:35.596841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.596804    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e852e734-c7b6-42a2-93b6-f7276bcedf63" path="/var/lib/kubelet/pods/e852e734-c7b6-42a2-93b6-f7276bcedf63/volumes"
Oct 31 20:13:35.886584 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.886547    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"]
Oct 31 20:13:35.887166 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.887141    5006 cpu_manager.go:414] "containerMap: removing container" podUID="75efbf1e-64e0-481d-b41c-1941975c5ff1" containerName="revoke-keys"
Oct 31 20:13:35.887166 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.887168    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75efbf1e-64e0-481d-b41c-1941975c5ff1" containerName="revoke-keys"
Oct 31 20:13:35.887335 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.887189    5006 cpu_manager.go:414] "containerMap: removing container" podUID="c751b999-02bd-4f36-814e-da70d5a2d94a" containerName="revoke-keys"
Oct 31 20:13:35.887335 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.887197    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c751b999-02bd-4f36-814e-da70d5a2d94a" containerName="revoke-keys"
Oct 31 20:13:35.887335 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.887208    5006 cpu_manager.go:414] "containerMap: removing container" podUID="41e920c5-f1ee-4aaa-ae85-88fa1e4aea58" containerName="payload-pre-processing"
Oct 31 20:13:35.887335 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.887217    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e920c5-f1ee-4aaa-ae85-88fa1e4aea58" containerName="payload-pre-processing"
Oct 31 20:13:35.887479 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.887346    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="75efbf1e-64e0-481d-b41c-1941975c5ff1" containerName="revoke-keys"
Oct 31 20:13:35.887479 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.887361    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c751b999-02bd-4f36-814e-da70d5a2d94a" containerName="revoke-keys"
Oct 31 20:13:35.887479 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.887371    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e920c5-f1ee-4aaa-ae85-88fa1e4aea58" containerName="payload-pre-processing"
Oct 31 20:13:35.892828 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.892805    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:35.943358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.943324    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"]
Oct 31 20:13:35.987015 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.986980    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9579efea-313c-40e9-8536-8893f36dc162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:35.987185 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.987036    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9579efea-313c-40e9-8536-8893f36dc162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:35.987185 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.987151    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxx4k\" (UniqueName: \"kubernetes.io/projected/9579efea-313c-40e9-8536-8893f36dc162-kube-api-access-pxx4k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:35.991216 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.991185    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:13:35.991431 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.991234    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:13:35.991431 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.991240    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:13:35.996143 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:35.996120    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:13:36.087842 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:36.087801    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9579efea-313c-40e9-8536-8893f36dc162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:36.088339 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:36.087856    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9579efea-313c-40e9-8536-8893f36dc162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:36.088339 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:36.087884    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxx4k\" (UniqueName: \"kubernetes.io/projected/9579efea-313c-40e9-8536-8893f36dc162-kube-api-access-pxx4k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:36.088740 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:36.088712    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9579efea-313c-40e9-8536-8893f36dc162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:36.090316 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:36.090295    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9579efea-313c-40e9-8536-8893f36dc162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:36.147447 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:36.147372    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxx4k\" (UniqueName: \"kubernetes.io/projected/9579efea-313c-40e9-8536-8893f36dc162-kube-api-access-pxx4k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:36.206487 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:36.206441    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:36.578528 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:13:36.578488    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9579efea_313c_40e9_8536_8893f36dc162.slice/crio-07cbf9c776f95588a378c49a421d76a4006bc4a197718da58d048076fbb6cf3d WatchSource:0}: Error finding container 07cbf9c776f95588a378c49a421d76a4006bc4a197718da58d048076fbb6cf3d: Status 404 returned error can't find the container with id 07cbf9c776f95588a378c49a421d76a4006bc4a197718da58d048076fbb6cf3d
Oct 31 20:13:36.580882 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:36.580852    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"]
Oct 31 20:13:36.845421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:36.845386    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs" event={"ID":"9579efea-313c-40e9-8536-8893f36dc162","Type":"ContainerStarted","Data":"07cbf9c776f95588a378c49a421d76a4006bc4a197718da58d048076fbb6cf3d"}
Oct 31 20:13:37.050574 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:37.050537    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"]
Oct 31 20:13:37.068956 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:37.068919    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c81605-lp6q7"]
Oct 31 20:13:37.598452 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:37.598420    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a1c6f3-45cd-4d90-8429-67fa43199198" path="/var/lib/kubelet/pods/52a1c6f3-45cd-4d90-8429-67fa43199198/volumes"
Oct 31 20:13:37.852818 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:37.852787    5006 generic.go:356] "Generic (PLEG): container finished" podID="9579efea-313c-40e9-8536-8893f36dc162" containerID="93a02ed4a8f1464a9bd60baeab8ac514e2cde352fc5c58cbdc504d033ad1f4ed" exitCode=0
Oct 31 20:13:37.853010 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:37.852873    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs" event={"ID":"9579efea-313c-40e9-8536-8893f36dc162","Type":"ContainerDied","Data":"93a02ed4a8f1464a9bd60baeab8ac514e2cde352fc5c58cbdc504d033ad1f4ed"}
Oct 31 20:13:38.030092 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:38.030058    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:38.056931 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:38.056900    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"]
Oct 31 20:13:38.059510 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:38.059482    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c92bd85d-l2wdj"]
Oct 31 20:13:38.284695 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:38.284566    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:39.006357 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.006326    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:39.015692 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.015661    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9579efea-313c-40e9-8536-8893f36dc162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9579efea-313c-40e9-8536-8893f36dc162-maas-api-openshift-ca\") pod \"9579efea-313c-40e9-8536-8893f36dc162\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") "
Oct 31 20:13:39.015825 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.015717    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9579efea-313c-40e9-8536-8893f36dc162-kube-api-access-pxx4k\" (UniqueName: \"kubernetes.io/projected/9579efea-313c-40e9-8536-8893f36dc162-kube-api-access-pxx4k\") pod \"9579efea-313c-40e9-8536-8893f36dc162\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") "
Oct 31 20:13:39.016018 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.015996    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9579efea-313c-40e9-8536-8893f36dc162-maas-api-openshift-ca" pod "9579efea-313c-40e9-8536-8893f36dc162" (UID: "9579efea-313c-40e9-8536-8893f36dc162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:13:39.017952 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.017920    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9579efea-313c-40e9-8536-8893f36dc162-kube-api-access-pxx4k" pod "9579efea-313c-40e9-8536-8893f36dc162" (UID: "9579efea-313c-40e9-8536-8893f36dc162"). InnerVolumeSpecName "kube-api-access-pxx4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:13:39.116508 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.116401    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9579efea-313c-40e9-8536-8893f36dc162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9579efea-313c-40e9-8536-8893f36dc162-maas-api-xks-ca\") pod \"9579efea-313c-40e9-8536-8893f36dc162\" (UID: \"9579efea-313c-40e9-8536-8893f36dc162\") "
Oct 31 20:13:39.116779 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.116753    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9579efea-313c-40e9-8536-8893f36dc162-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:39.116779 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.116777    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxx4k\" (UniqueName: \"kubernetes.io/projected/9579efea-313c-40e9-8536-8893f36dc162-kube-api-access-pxx4k\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:39.118868 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.118841    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9579efea-313c-40e9-8536-8893f36dc162-maas-api-xks-ca" pod "9579efea-313c-40e9-8536-8893f36dc162" (UID: "9579efea-313c-40e9-8536-8893f36dc162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:13:39.217491 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.217443    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9579efea-313c-40e9-8536-8893f36dc162-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:39.387153 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.387043    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:39.597379 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.597345    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9327afce-2584-4b26-97a8-85cc442e6bc2" path="/var/lib/kubelet/pods/9327afce-2584-4b26-97a8-85cc442e6bc2/volumes"
Oct 31 20:13:39.865556 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.865523    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"
Oct 31 20:13:39.865768 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.865554    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs" event={"ID":"9579efea-313c-40e9-8536-8893f36dc162","Type":"ContainerDied","Data":"07cbf9c776f95588a378c49a421d76a4006bc4a197718da58d048076fbb6cf3d"}
Oct 31 20:13:39.865768 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:39.865643    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07cbf9c776f95588a378c49a421d76a4006bc4a197718da58d048076fbb6cf3d"
Oct 31 20:13:40.744290 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:40.744251    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:40.985809 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:40.985777    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:41.211499 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:41.211460    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:44.370669 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:44.370604    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:44.822869 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:44.822776    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:13:47.822224 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:47.822188    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:48.036081 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:48.036018    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:49.279971 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:49.279932    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:54.089606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:54.089534    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:58.473421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:13:58.473362    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:00.045358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:00.045323    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:00.275952 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:00.275892    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:00.508639 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:00.508600    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"]
Oct 31 20:14:00.558043 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:00.558010    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6eeb83-hmtxs"]
Oct 31 20:14:00.621917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:00.621855    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:00.976042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:00.975991    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:01.597804 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:01.597766    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9579efea-313c-40e9-8536-8893f36dc162" path="/var/lib/kubelet/pods/9579efea-313c-40e9-8536-8893f36dc162/volumes"
Oct 31 20:14:03.618660 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:03.618611    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:14:04.655664 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:04.655598    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:04.877075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:04.877014    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:05.394163 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:14:05.394122    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:14:05.579775 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:05.579720    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:06.773508 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:06.773462    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:07.052039 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:07.051951    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:07.272785 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:07.272750    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:07.528552 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:07.528514    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:10.191645 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:10.191598    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"]
Oct 31 20:14:10.192160 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:10.191907    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy" containerID="cri-o://0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a" gracePeriod=30
Oct 31 20:14:10.241730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:10.241693    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"]
Oct 31 20:14:10.241946 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:10.241922    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy" containerID="cri-o://c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f" gracePeriod=30
Oct 31 20:14:10.733793 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:10.733757    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:12.418985 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:12.418945    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"]
Oct 31 20:14:12.433504 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:12.433464    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"]
Oct 31 20:14:12.595992 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:12.595959    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:15.535558 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.535524    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:14:15.538621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.538594    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:14:15.693834 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.693760    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-podinfo\") pod \"d71847a6-27eb-4639-abba-6c9e62a0e15a\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") "
Oct 31 20:14:15.694042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.693843    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-data\") pod \"d71847a6-27eb-4639-abba-6c9e62a0e15a\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") "
Oct 31 20:14:15.694042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.693880    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-podinfo\") pod \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") "
Oct 31 20:14:15.694042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.693927    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-certs\") pod \"d71847a6-27eb-4639-abba-6c9e62a0e15a\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") "
Oct 31 20:14:15.694042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.693961    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-socket\") pod \"d71847a6-27eb-4639-abba-6c9e62a0e15a\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") "
Oct 31 20:14:15.694042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694018    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-kube-api-access-pk85l\" (UniqueName: \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-kube-api-access-pk85l\") pod \"d71847a6-27eb-4639-abba-6c9e62a0e15a\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") "
Oct 31 20:14:15.694319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694070    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-credential-socket\") pod \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") "
Oct 31 20:14:15.694319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694159    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-token\" (UniqueName: \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-token\") pod \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") "
Oct 31 20:14:15.694319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694192    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-credential-socket\") pod \"d71847a6-27eb-4639-abba-6c9e62a0e15a\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") "
Oct 31 20:14:15.694319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694237    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-data\") pod \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") "
Oct 31 20:14:15.694319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694286    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-kube-api-access-w6fx6\" (UniqueName: \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-kube-api-access-w6fx6\") pod \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") "
Oct 31 20:14:15.694557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694320    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71847a6-27eb-4639-abba-6c9e62a0e15a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71847a6-27eb-4639-abba-6c9e62a0e15a-istiod-ca-cert\") pod \"d71847a6-27eb-4639-abba-6c9e62a0e15a\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") "
Oct 31 20:14:15.694557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694370    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-envoy\") pod \"d71847a6-27eb-4639-abba-6c9e62a0e15a\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") "
Oct 31 20:14:15.694557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694407    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istiod-ca-cert\") pod \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") "
Oct 31 20:14:15.694557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694442    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-token\" (UniqueName: \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-token\") pod \"d71847a6-27eb-4639-abba-6c9e62a0e15a\" (UID: \"d71847a6-27eb-4639-abba-6c9e62a0e15a\") "
Oct 31 20:14:15.694557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694476    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-envoy\") pod \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") "
Oct 31 20:14:15.694557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694509    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-certs\") pod \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") "
Oct 31 20:14:15.694557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694544    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-socket\") pod \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\" (UID: \"47f432dd-3da5-4865-99be-9cd2cb7be3a4\") "
Oct 31 20:14:15.695265 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.694967    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-credential-socket" pod "d71847a6-27eb-4639-abba-6c9e62a0e15a" (UID: "d71847a6-27eb-4639-abba-6c9e62a0e15a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:15.695265 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.695196    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-data" pod "d71847a6-27eb-4639-abba-6c9e62a0e15a" (UID: "d71847a6-27eb-4639-abba-6c9e62a0e15a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:15.695432 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.695334    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71847a6-27eb-4639-abba-6c9e62a0e15a-istiod-ca-cert" pod "d71847a6-27eb-4639-abba-6c9e62a0e15a" (UID: "d71847a6-27eb-4639-abba-6c9e62a0e15a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:14:15.696115 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.695974    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-certs" pod "d71847a6-27eb-4639-abba-6c9e62a0e15a" (UID: "d71847a6-27eb-4639-abba-6c9e62a0e15a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:15.696223 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.696165    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-socket" pod "47f432dd-3da5-4865-99be-9cd2cb7be3a4" (UID: "47f432dd-3da5-4865-99be-9cd2cb7be3a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:15.696999 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.696378    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-data" pod "47f432dd-3da5-4865-99be-9cd2cb7be3a4" (UID: "47f432dd-3da5-4865-99be-9cd2cb7be3a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:15.696999 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.696578    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-credential-socket" pod "47f432dd-3da5-4865-99be-9cd2cb7be3a4" (UID: "47f432dd-3da5-4865-99be-9cd2cb7be3a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:15.696999 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.696601    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-certs" pod "47f432dd-3da5-4865-99be-9cd2cb7be3a4" (UID: "47f432dd-3da5-4865-99be-9cd2cb7be3a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:15.696999 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.696726    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-podinfo" pod "d71847a6-27eb-4639-abba-6c9e62a0e15a" (UID: "d71847a6-27eb-4639-abba-6c9e62a0e15a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:14:15.696999 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.696770    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istiod-ca-cert" pod "47f432dd-3da5-4865-99be-9cd2cb7be3a4" (UID: "47f432dd-3da5-4865-99be-9cd2cb7be3a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:14:15.696999 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.696951    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-socket" pod "d71847a6-27eb-4639-abba-6c9e62a0e15a" (UID: "d71847a6-27eb-4639-abba-6c9e62a0e15a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:15.697938 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.697912    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-kube-api-access-pk85l" pod "d71847a6-27eb-4639-abba-6c9e62a0e15a" (UID: "d71847a6-27eb-4639-abba-6c9e62a0e15a"). InnerVolumeSpecName "kube-api-access-pk85l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:14:15.698514 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.698489    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-envoy" pod "d71847a6-27eb-4639-abba-6c9e62a0e15a" (UID: "d71847a6-27eb-4639-abba-6c9e62a0e15a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:15.698630 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.698492    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-envoy" pod "47f432dd-3da5-4865-99be-9cd2cb7be3a4" (UID: "47f432dd-3da5-4865-99be-9cd2cb7be3a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:15.698843 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.698818    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-podinfo" pod "47f432dd-3da5-4865-99be-9cd2cb7be3a4" (UID: "47f432dd-3da5-4865-99be-9cd2cb7be3a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:14:15.699175 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.699152    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-token" pod "d71847a6-27eb-4639-abba-6c9e62a0e15a" (UID: "d71847a6-27eb-4639-abba-6c9e62a0e15a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:14:15.699289 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.699271    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-token" pod "47f432dd-3da5-4865-99be-9cd2cb7be3a4" (UID: "47f432dd-3da5-4865-99be-9cd2cb7be3a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:14:15.699866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.699851    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-kube-api-access-w6fx6" pod "47f432dd-3da5-4865-99be-9cd2cb7be3a4" (UID: "47f432dd-3da5-4865-99be-9cd2cb7be3a4"). InnerVolumeSpecName "kube-api-access-w6fx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:14:15.796312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796202    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796249    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796267    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796283    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796296    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796310    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796324    5006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-podinfo\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796340    5006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796355    5006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-workload-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796369    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk85l\" (UniqueName: \"kubernetes.io/projected/d71847a6-27eb-4639-abba-6c9e62a0e15a-kube-api-access-pk85l\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796383    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796423    5006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-token\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796437    5006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-credential-socket\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796452    5006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istio-data\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796466    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6fx6\" (UniqueName: \"kubernetes.io/projected/47f432dd-3da5-4865-99be-9cd2cb7be3a4-kube-api-access-w6fx6\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796478    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71847a6-27eb-4639-abba-6c9e62a0e15a-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796491    5006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71847a6-27eb-4639-abba-6c9e62a0e15a-istio-envoy\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.796832 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.796503    5006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47f432dd-3da5-4865-99be-9cd2cb7be3a4-istiod-ca-cert\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:15.806062 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:15.805962    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:16.088390 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.088359    5006 generic.go:356] "Generic (PLEG): container finished" podID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerID="c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f" exitCode=0
Oct 31 20:14:16.088579 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.088401    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"
Oct 31 20:14:16.088579 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.088449    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" event={"ID":"47f432dd-3da5-4865-99be-9cd2cb7be3a4","Type":"ContainerDied","Data":"c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f"}
Oct 31 20:14:16.088579 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.088505    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f" event={"ID":"47f432dd-3da5-4865-99be-9cd2cb7be3a4","Type":"ContainerDied","Data":"888bbaf55c02f5711088578de116f130432e6cab21a63afa25cb11cac7730355"}
Oct 31 20:14:16.088579 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.088531    5006 scope.go:122] "RemoveContainer" containerID="c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f"
Oct 31 20:14:16.090249 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.090210    5006 generic.go:356] "Generic (PLEG): container finished" podID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerID="0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a" exitCode=0
Oct 31 20:14:16.090373 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.090265    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"
Oct 31 20:14:16.090373 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.090271    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" event={"ID":"d71847a6-27eb-4639-abba-6c9e62a0e15a","Type":"ContainerDied","Data":"0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a"}
Oct 31 20:14:16.090373 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.090317    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m" event={"ID":"d71847a6-27eb-4639-abba-6c9e62a0e15a","Type":"ContainerDied","Data":"9d7ba09d5e01eecb2c09a0a7136026665ee57931b589553baade1d86bbf8ce15"}
Oct 31 20:14:16.099204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.099185    5006 scope.go:122] "RemoveContainer" containerID="c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f"
Oct 31 20:14:16.099583 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:14:16.099553    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f\": container with ID starting with c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f not found: ID does not exist" containerID="c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f"
Oct 31 20:14:16.099694 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.099595    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f"} err="failed to get container status \"c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f\": rpc error: code = NotFound desc = could not find container \"c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f\": container with ID starting with c906480771ae08521c24753e651b5d30d8582cfe2244d1e3ac04c21e4842855f not found: ID does not exist"
Oct 31 20:14:16.099694 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.099615    5006 scope.go:122] "RemoveContainer" containerID="0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a"
Oct 31 20:14:16.109092 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.109068    5006 scope.go:122] "RemoveContainer" containerID="0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a"
Oct 31 20:14:16.109437 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:14:16.109411    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a\": container with ID starting with 0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a not found: ID does not exist" containerID="0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a"
Oct 31 20:14:16.109541 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.109447    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a"} err="failed to get container status \"0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a\": rpc error: code = NotFound desc = could not find container \"0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a\": container with ID starting with 0720363e610a3432882cd1ebb7072a5a8f5f33c70fb676609372f48da7a5c93a not found: ID does not exist"
Oct 31 20:14:16.255243 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.255206    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"]
Oct 31 20:14:16.304995 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.304961    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zf78f"]
Oct 31 20:14:16.418203 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.418111    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"]
Oct 31 20:14:16.422902 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:16.422875    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gjb7m"]
Oct 31 20:14:17.597667 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:17.597616    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" path="/var/lib/kubelet/pods/47f432dd-3da5-4865-99be-9cd2cb7be3a4/volumes"
Oct 31 20:14:17.598112 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:17.598097    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" path="/var/lib/kubelet/pods/d71847a6-27eb-4639-abba-6c9e62a0e15a/volumes"
Oct 31 20:14:18.447785 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:18.447751    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:19.925314 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:19.925279    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:20.039569 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.039535    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"]
Oct 31 20:14:20.039990 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.039975    5006 cpu_manager.go:414] "containerMap: removing container" podUID="9579efea-313c-40e9-8536-8893f36dc162" containerName="revoke-keys"
Oct 31 20:14:20.040047 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.039993    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9579efea-313c-40e9-8536-8893f36dc162" containerName="revoke-keys"
Oct 31 20:14:20.040047 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.040019    5006 cpu_manager.go:414] "containerMap: removing container" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy"
Oct 31 20:14:20.040047 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.040024    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy"
Oct 31 20:14:20.040047 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.040032    5006 cpu_manager.go:414] "containerMap: removing container" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy"
Oct 31 20:14:20.040047 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.040037    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy"
Oct 31 20:14:20.040224 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.040095    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71847a6-27eb-4639-abba-6c9e62a0e15a" containerName="istio-proxy"
Oct 31 20:14:20.040224 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.040104    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f432dd-3da5-4865-99be-9cd2cb7be3a4" containerName="istio-proxy"
Oct 31 20:14:20.040224 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.040114    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9579efea-313c-40e9-8536-8893f36dc162" containerName="revoke-keys"
Oct 31 20:14:20.044885 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.044862    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.096454 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.096418    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"]
Oct 31 20:14:20.137355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.137319    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjfx\" (UniqueName: \"kubernetes.io/projected/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-kube-api-access-tmjfx\") pod \"maas-api-revoke-sub-9eaee8bf-xsv68\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.137532 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.137375    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eaee8bf-xsv68\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.137532 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.137479    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eaee8bf-xsv68\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.137972 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.137951    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:20.143910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.143885    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:14:20.143910 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.143895    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:14:20.144069 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.143895    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:14:20.147704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.147685    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:14:20.238055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.237964    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eaee8bf-xsv68\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.238055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.238020    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eaee8bf-xsv68\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.238267 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.238081    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjfx\" (UniqueName: \"kubernetes.io/projected/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-kube-api-access-tmjfx\") pod \"maas-api-revoke-sub-9eaee8bf-xsv68\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.238838 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.238815    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eaee8bf-xsv68\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.240491 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.240474    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eaee8bf-xsv68\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.295580 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.295536    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjfx\" (UniqueName: \"kubernetes.io/projected/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-kube-api-access-tmjfx\") pod \"maas-api-revoke-sub-9eaee8bf-xsv68\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.360729 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.360688    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:20.731087 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:14:20.731055    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e6ff55_5c2a_42a1_808d_2d85cdd291ff.slice/crio-ed74bdf5582cfdf760141adb840ee63f2c47c3d00e457d5c0873d5c51450c6be WatchSource:0}: Error finding container ed74bdf5582cfdf760141adb840ee63f2c47c3d00e457d5c0873d5c51450c6be: Status 404 returned error can't find the container with id ed74bdf5582cfdf760141adb840ee63f2c47c3d00e457d5c0873d5c51450c6be
Oct 31 20:14:20.732804 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:20.732776    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"]
Oct 31 20:14:21.124598 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:21.124547    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68" event={"ID":"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff","Type":"ContainerStarted","Data":"024cf633a39e88bb6bb3a09596bc4fef876345518d76f43f62b2a6e1c3cd866a"}
Oct 31 20:14:21.125149 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:21.124677    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68" event={"ID":"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff","Type":"ContainerStarted","Data":"ed74bdf5582cfdf760141adb840ee63f2c47c3d00e457d5c0873d5c51450c6be"}
Oct 31 20:14:21.284539 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:21.284430    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68" podStartSLOduration=2.284416205 podStartE2EDuration="2.284416205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:14:21.239707551 +0000 UTC m=+3479.769241659" watchObservedRunningTime="2026-10-31 20:14:21.284416205 +0000 UTC m=+3479.813950250"
Oct 31 20:14:22.135887 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:22.135850    5006 generic.go:356] "Generic (PLEG): container finished" podID="b2e6ff55-5c2a-42a1-808d-2d85cdd291ff" containerID="024cf633a39e88bb6bb3a09596bc4fef876345518d76f43f62b2a6e1c3cd866a" exitCode=0
Oct 31 20:14:22.136358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:22.135931    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68" event={"ID":"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff","Type":"ContainerDied","Data":"024cf633a39e88bb6bb3a09596bc4fef876345518d76f43f62b2a6e1c3cd866a"}
Oct 31 20:14:23.288629 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:23.288595    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:23.372739 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:23.372696    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-openshift-ca\") pod \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") "
Oct 31 20:14:23.372957 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:23.372828    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-kube-api-access-tmjfx\" (UniqueName: \"kubernetes.io/projected/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-kube-api-access-tmjfx\") pod \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") "
Oct 31 20:14:23.372957 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:23.372861    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-xks-ca\") pod \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\" (UID: \"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff\") "
Oct 31 20:14:23.373085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:23.373055    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-openshift-ca" pod "b2e6ff55-5c2a-42a1-808d-2d85cdd291ff" (UID: "b2e6ff55-5c2a-42a1-808d-2d85cdd291ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:14:23.373204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:23.373189    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:23.375188 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:23.375154    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-xks-ca" pod "b2e6ff55-5c2a-42a1-808d-2d85cdd291ff" (UID: "b2e6ff55-5c2a-42a1-808d-2d85cdd291ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:14:23.375188 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:23.375167    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-kube-api-access-tmjfx" pod "b2e6ff55-5c2a-42a1-808d-2d85cdd291ff" (UID: "b2e6ff55-5c2a-42a1-808d-2d85cdd291ff"). InnerVolumeSpecName "kube-api-access-tmjfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:14:23.474805 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:23.474703    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmjfx\" (UniqueName: \"kubernetes.io/projected/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-kube-api-access-tmjfx\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:23.474805 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:23.474747    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:24.149634 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:24.149602    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"
Oct 31 20:14:24.149864 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:24.149631    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68" event={"ID":"b2e6ff55-5c2a-42a1-808d-2d85cdd291ff","Type":"ContainerDied","Data":"ed74bdf5582cfdf760141adb840ee63f2c47c3d00e457d5c0873d5c51450c6be"}
Oct 31 20:14:24.149864 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:24.149711    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed74bdf5582cfdf760141adb840ee63f2c47c3d00e457d5c0873d5c51450c6be"
Oct 31 20:14:24.908892 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:24.908859    5006 scope.go:122] "RemoveContainer" containerID="7c867c30a153733deea84e14817f8167c4b55766eb5c238a2e57cf5bc83ed21f"
Oct 31 20:14:24.918985 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:24.918947    5006 scope.go:122] "RemoveContainer" containerID="c62e2fb0dc3d980cc527a30523317a94ab4a295feba27eef0e692a95b8c7c0cc"
Oct 31 20:14:27.440263 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.440210    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main" containerID="cri-o://e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b" gracePeriod=45
Oct 31 20:14:27.453951 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.453910    5006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main" containerID="cri-o://55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d" gracePeriod=45
Oct 31 20:14:27.730787 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.730759    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 20:14:27.745977 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.745946    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 20:14:27.817292 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817232    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-dshm\") pod \"71f16468-0585-46c2-a02d-09a958fc961f\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") "
Oct 31 20:14:27.817482 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817331    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-home\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-home\") pod \"9c78a101-f486-45ac-a45b-e4ba2366fe68\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") "
Oct 31 20:14:27.817482 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817381    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-model-cache\") pod \"9c78a101-f486-45ac-a45b-e4ba2366fe68\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") "
Oct 31 20:14:27.817482 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817418    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-home\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-home\") pod \"71f16468-0585-46c2-a02d-09a958fc961f\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") "
Oct 31 20:14:27.817482 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817450    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-tmp-dir\") pod \"9c78a101-f486-45ac-a45b-e4ba2366fe68\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") "
Oct 31 20:14:27.817482 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817477    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c78a101-f486-45ac-a45b-e4ba2366fe68-kube-api-access-wktgc\" (UniqueName: \"kubernetes.io/projected/9c78a101-f486-45ac-a45b-e4ba2366fe68-kube-api-access-wktgc\") pod \"9c78a101-f486-45ac-a45b-e4ba2366fe68\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") "
Oct 31 20:14:27.817817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817521    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-tmp-dir\") pod \"71f16468-0585-46c2-a02d-09a958fc961f\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") "
Oct 31 20:14:27.817817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817580    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c78a101-f486-45ac-a45b-e4ba2366fe68-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c78a101-f486-45ac-a45b-e4ba2366fe68-tls-certs\") pod \"9c78a101-f486-45ac-a45b-e4ba2366fe68\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") "
Oct 31 20:14:27.817817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817597    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-home" pod "9c78a101-f486-45ac-a45b-e4ba2366fe68" (UID: "9c78a101-f486-45ac-a45b-e4ba2366fe68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:27.817817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817613    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71f16468-0585-46c2-a02d-09a958fc961f-tls-certs\" (UniqueName: \"kubernetes.io/secret/71f16468-0585-46c2-a02d-09a958fc961f-tls-certs\") pod \"71f16468-0585-46c2-a02d-09a958fc961f\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") "
Oct 31 20:14:27.817817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817736    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-dshm\") pod \"9c78a101-f486-45ac-a45b-e4ba2366fe68\" (UID: \"9c78a101-f486-45ac-a45b-e4ba2366fe68\") "
Oct 31 20:14:27.817817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817779    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-model-cache\") pod \"71f16468-0585-46c2-a02d-09a958fc961f\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") "
Oct 31 20:14:27.817817 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817814    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f16468-0585-46c2-a02d-09a958fc961f-kube-api-access-dxz2w\" (UniqueName: \"kubernetes.io/projected/71f16468-0585-46c2-a02d-09a958fc961f-kube-api-access-dxz2w\") pod \"71f16468-0585-46c2-a02d-09a958fc961f\" (UID: \"71f16468-0585-46c2-a02d-09a958fc961f\") "
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817815    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-model-cache" pod "9c78a101-f486-45ac-a45b-e4ba2366fe68" (UID: "9c78a101-f486-45ac-a45b-e4ba2366fe68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817821    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-tmp-dir" pod "9c78a101-f486-45ac-a45b-e4ba2366fe68" (UID: "9c78a101-f486-45ac-a45b-e4ba2366fe68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.817796    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-home" pod "71f16468-0585-46c2-a02d-09a958fc961f" (UID: "71f16468-0585-46c2-a02d-09a958fc961f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.818035    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-model-cache" pod "71f16468-0585-46c2-a02d-09a958fc961f" (UID: "71f16468-0585-46c2-a02d-09a958fc961f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.818048    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-tmp-dir" pod "71f16468-0585-46c2-a02d-09a958fc961f" (UID: "71f16468-0585-46c2-a02d-09a958fc961f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.818191    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.818212    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.818226    5006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-home\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.818240    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.818252    5006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-tmp-dir\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.818264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.818264    5006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-model-cache\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.820144 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.820116    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c78a101-f486-45ac-a45b-e4ba2366fe68-tls-certs" pod "9c78a101-f486-45ac-a45b-e4ba2366fe68" (UID: "9c78a101-f486-45ac-a45b-e4ba2366fe68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:14:27.820294 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.820273    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-dshm" pod "71f16468-0585-46c2-a02d-09a958fc961f" (UID: "71f16468-0585-46c2-a02d-09a958fc961f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:27.820355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.820314    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f16468-0585-46c2-a02d-09a958fc961f-tls-certs" pod "71f16468-0585-46c2-a02d-09a958fc961f" (UID: "71f16468-0585-46c2-a02d-09a958fc961f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:14:27.820355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.820323    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-dshm" pod "9c78a101-f486-45ac-a45b-e4ba2366fe68" (UID: "9c78a101-f486-45ac-a45b-e4ba2366fe68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:14:27.820355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.820323    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c78a101-f486-45ac-a45b-e4ba2366fe68-kube-api-access-wktgc" pod "9c78a101-f486-45ac-a45b-e4ba2366fe68" (UID: "9c78a101-f486-45ac-a45b-e4ba2366fe68"). InnerVolumeSpecName "kube-api-access-wktgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:14:27.820488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.820395    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f16468-0585-46c2-a02d-09a958fc961f-kube-api-access-dxz2w" pod "71f16468-0585-46c2-a02d-09a958fc961f" (UID: "71f16468-0585-46c2-a02d-09a958fc961f"). InnerVolumeSpecName "kube-api-access-dxz2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:14:27.919662 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.919595    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wktgc\" (UniqueName: \"kubernetes.io/projected/9c78a101-f486-45ac-a45b-e4ba2366fe68-kube-api-access-wktgc\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.919662 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.919644    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c78a101-f486-45ac-a45b-e4ba2366fe68-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.919896 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.919686    5006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f16468-0585-46c2-a02d-09a958fc961f-tls-certs\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.919896 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.919695    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c78a101-f486-45ac-a45b-e4ba2366fe68-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.919896 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.919705    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxz2w\" (UniqueName: \"kubernetes.io/projected/71f16468-0585-46c2-a02d-09a958fc961f-kube-api-access-dxz2w\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:27.919896 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:27.919713    5006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f16468-0585-46c2-a02d-09a958fc961f-dshm\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:14:28.173421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.173386    5006 generic.go:356] "Generic (PLEG): container finished" podID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerID="e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b" exitCode=0
Oct 31 20:14:28.173600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.173433    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"
Oct 31 20:14:28.173600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.173473    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" event={"ID":"9c78a101-f486-45ac-a45b-e4ba2366fe68","Type":"ContainerDied","Data":"e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b"}
Oct 31 20:14:28.173600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.173519    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh" event={"ID":"9c78a101-f486-45ac-a45b-e4ba2366fe68","Type":"ContainerDied","Data":"55f5dfed265a2539fb58c2c3ab9b843a5ea02448852a0c1646c89acc237e7591"}
Oct 31 20:14:28.173600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.173537    5006 scope.go:122] "RemoveContainer" containerID="e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b"
Oct 31 20:14:28.175122 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.175098    5006 generic.go:356] "Generic (PLEG): container finished" podID="71f16468-0585-46c2-a02d-09a958fc961f" containerID="55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d" exitCode=0
Oct 31 20:14:28.175233 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.175155    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"
Oct 31 20:14:28.175233 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.175164    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" event={"ID":"71f16468-0585-46c2-a02d-09a958fc961f","Type":"ContainerDied","Data":"55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d"}
Oct 31 20:14:28.175233 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.175203    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v" event={"ID":"71f16468-0585-46c2-a02d-09a958fc961f","Type":"ContainerDied","Data":"4a3f22e70b879fc03a07df30fa8c5289c2ace77e23f8c718919f03a6ffb9ca6e"}
Oct 31 20:14:28.184288 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.184262    5006 scope.go:122] "RemoveContainer" containerID="e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b"
Oct 31 20:14:28.184724 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:14:28.184692    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b\": container with ID starting with e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b not found: ID does not exist" containerID="e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b"
Oct 31 20:14:28.184814 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.184732    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b"} err="failed to get container status \"e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b\": rpc error: code = NotFound desc = could not find container \"e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b\": container with ID starting with e623b30c952d89dcc47cb20cbac99915b2246f52969a1c98851d543b86270a6b not found: ID does not exist"
Oct 31 20:14:28.184814 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.184758    5006 scope.go:122] "RemoveContainer" containerID="55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d"
Oct 31 20:14:28.193840 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.193817    5006 scope.go:122] "RemoveContainer" containerID="55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d"
Oct 31 20:14:28.194140 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:14:28.194116    5006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d\": container with ID starting with 55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d not found: ID does not exist" containerID="55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d"
Oct 31 20:14:28.194215 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.194147    5006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d"} err="failed to get container status \"55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d\": rpc error: code = NotFound desc = could not find container \"55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d\": container with ID starting with 55ac2a7025b53551792472ace7da158e6c46616d31590dc2dc4c7dcfc1548f0d not found: ID does not exist"
Oct 31 20:14:28.339274 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.339229    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"]
Oct 31 20:14:28.389932 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.389887    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-w0-093b97-kserve-5c86b9cf4c-gt2sh"]
Oct 31 20:14:28.499558 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.499520    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"]
Oct 31 20:14:28.502808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:28.502778    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/simulator-w0-093b97-kserve-8446dcf765-th55v"]
Oct 31 20:14:29.597177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:29.597143    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f16468-0585-46c2-a02d-09a958fc961f" path="/var/lib/kubelet/pods/71f16468-0585-46c2-a02d-09a958fc961f/volumes"
Oct 31 20:14:29.597718 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:29.597549    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" path="/var/lib/kubelet/pods/9c78a101-f486-45ac-a45b-e4ba2366fe68/volumes"
Oct 31 20:14:31.362768 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:31.362725    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:34.992490 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:34.992446    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:35.322401 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:35.322312    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:14:35.394227 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:14:35.394190    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:14:36.552018 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:36.551975    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:36.824266 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:36.824167    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:37.486112 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:37.486072    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:37.710362 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:37.710321    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:43.263556 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:43.263515    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:43.500391 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:43.500359    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:43.735430 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:43.735394    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:49.795945 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:49.795905    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:50.036897 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:50.036864    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:50.249468 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:50.249434    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:51.452256 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:51.452217    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:14:55.610043 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:55.610007    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:55.829956 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:14:55.829922    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:05.393951 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:15:05.393903    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:15:08.054514 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:08.054472    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"]
Oct 31 20:15:08.104353 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:08.104321    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52950dab-zmwbz"]
Oct 31 20:15:09.597020 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:09.596987    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b585104a-8928-416a-a03f-f612804c8fa5" path="/var/lib/kubelet/pods/b585104a-8928-416a-a03f-f612804c8fa5/volumes"
Oct 31 20:15:10.652994 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.652956    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"]
Oct 31 20:15:10.653406 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.653377    5006 cpu_manager.go:414] "containerMap: removing container" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main"
Oct 31 20:15:10.653406 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.653390    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main"
Oct 31 20:15:10.653406 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.653403    5006 cpu_manager.go:414] "containerMap: removing container" podUID="b2e6ff55-5c2a-42a1-808d-2d85cdd291ff" containerName="revoke-keys"
Oct 31 20:15:10.653516 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.653408    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e6ff55-5c2a-42a1-808d-2d85cdd291ff" containerName="revoke-keys"
Oct 31 20:15:10.653516 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.653434    5006 cpu_manager.go:414] "containerMap: removing container" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main"
Oct 31 20:15:10.653516 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.653439    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main"
Oct 31 20:15:10.653516 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.653500    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f16468-0585-46c2-a02d-09a958fc961f" containerName="main"
Oct 31 20:15:10.653516 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.653508    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e6ff55-5c2a-42a1-808d-2d85cdd291ff" containerName="revoke-keys"
Oct 31 20:15:10.653516 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.653517    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c78a101-f486-45ac-a45b-e4ba2366fe68" containerName="main"
Oct 31 20:15:10.657055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.657035    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.722336 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.722297    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdpjr\" (UniqueName: \"kubernetes.io/projected/421392b6-59fb-4d6d-a35f-acfd09346096-kube-api-access-mdpjr\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.722336 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.722337    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.722558 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.722357    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.722558 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.722442    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.722558 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.722537    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.722704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.722577    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421392b6-59fb-4d6d-a35f-acfd09346096-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.722704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.722646    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421392b6-59fb-4d6d-a35f-acfd09346096-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.722704 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.722698    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421392b6-59fb-4d6d-a35f-acfd09346096-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.722812 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.722719    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.762741 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.762702    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"]
Oct 31 20:15:10.823694 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.823634    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdpjr\" (UniqueName: \"kubernetes.io/projected/421392b6-59fb-4d6d-a35f-acfd09346096-kube-api-access-mdpjr\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.823888 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.823705    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.823888 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.823723    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.823888 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.823742    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.823888 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.823760    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.823888 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.823780    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421392b6-59fb-4d6d-a35f-acfd09346096-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.823888 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.823809    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421392b6-59fb-4d6d-a35f-acfd09346096-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.824211 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.824005    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421392b6-59fb-4d6d-a35f-acfd09346096-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.824211 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.824055    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.824211 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.824205    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.824387 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.824239    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.824387 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.824295    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.824492 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.824382    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.824609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.824588    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421392b6-59fb-4d6d-a35f-acfd09346096-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.826407 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.826389    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421392b6-59fb-4d6d-a35f-acfd09346096-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.826672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.826634    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421392b6-59fb-4d6d-a35f-acfd09346096-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.881475 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.881445    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421392b6-59fb-4d6d-a35f-acfd09346096-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.881674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.881625    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdpjr\" (UniqueName: \"kubernetes.io/projected/421392b6-59fb-4d6d-a35f-acfd09346096-kube-api-access-mdpjr\") pod \"maas-default-gateway-openshift-default-58d898bd55-vqn2k\" (UID: \"421392b6-59fb-4d6d-a35f-acfd09346096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:10.970674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:10.970557    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:11.385396 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:15:11.385360    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421392b6_59fb_4d6d_a35f_acfd09346096.slice/crio-301467a662bbd70afc4efe01fa14cf87297a04377a56b66b9b5c2ce22beb2dcf WatchSource:0}: Error finding container 301467a662bbd70afc4efe01fa14cf87297a04377a56b66b9b5c2ce22beb2dcf: Status 404 returned error can't find the container with id 301467a662bbd70afc4efe01fa14cf87297a04377a56b66b9b5c2ce22beb2dcf
Oct 31 20:15:11.387766 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:11.387731    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:15:11.387917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:11.387800    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:15:11.387917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:11.387755    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"]
Oct 31 20:15:11.387917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:11.387830    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:15:11.423427 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:11.423396    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" event={"ID":"421392b6-59fb-4d6d-a35f-acfd09346096","Type":"ContainerStarted","Data":"301467a662bbd70afc4efe01fa14cf87297a04377a56b66b9b5c2ce22beb2dcf"}
Oct 31 20:15:12.181464 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.181426    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"]
Oct 31 20:15:12.185371 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.185345    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.239373 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.239339    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:15:12.239565 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.239432    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:15:12.239675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.239642    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:15:12.239771 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.239759    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:15:12.240407 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.240382    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-td2q5\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.240524 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.240464    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvqv\" (UniqueName: \"kubernetes.io/projected/2ae1a099-1503-4ccf-a8a2-611bf5acec79-kube-api-access-fbvqv\") pod \"maas-api-revoke-sub-85237e45-td2q5\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.240585 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.240550    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-td2q5\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.240585 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.240551    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"]
Oct 31 20:15:12.342006 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.341973    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-td2q5\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.342195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.342036    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-td2q5\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.342195 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.342077    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvqv\" (UniqueName: \"kubernetes.io/projected/2ae1a099-1503-4ccf-a8a2-611bf5acec79-kube-api-access-fbvqv\") pod \"maas-api-revoke-sub-85237e45-td2q5\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.342745 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.342723    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-td2q5\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.344423 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.344403    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-td2q5\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.421570 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.421529    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvqv\" (UniqueName: \"kubernetes.io/projected/2ae1a099-1503-4ccf-a8a2-611bf5acec79-kube-api-access-fbvqv\") pod \"maas-api-revoke-sub-85237e45-td2q5\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.431444 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.431410    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" event={"ID":"421392b6-59fb-4d6d-a35f-acfd09346096","Type":"ContainerStarted","Data":"69825be6733c1572a2e5a6a39954b1be181f840d62e58884d8acf015755abc7a"}
Oct 31 20:15:12.523042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.522953    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:12.549731 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.549683    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podStartSLOduration=2.549667792 podStartE2EDuration="2.549667792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:12.548469297 +0000 UTC m=+3531.078003362" watchObservedRunningTime="2026-10-31 20:15:12.549667792 +0000 UTC m=+3531.079201841"
Oct 31 20:15:12.890962 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:15:12.890932    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae1a099_1503_4ccf_a8a2_611bf5acec79.slice/crio-9057e951ed5b7f1f2eeae7ef5200b975764c071374cedb38bbf6b872ec388526 WatchSource:0}: Error finding container 9057e951ed5b7f1f2eeae7ef5200b975764c071374cedb38bbf6b872ec388526: Status 404 returned error can't find the container with id 9057e951ed5b7f1f2eeae7ef5200b975764c071374cedb38bbf6b872ec388526
Oct 31 20:15:12.893759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.893660    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"]
Oct 31 20:15:12.971464 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.971430    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:12.973148 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.973121    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 31 20:15:12.973271 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:12.973173    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 31 20:15:13.439010 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:13.438975    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5" event={"ID":"2ae1a099-1503-4ccf-a8a2-611bf5acec79","Type":"ContainerStarted","Data":"db46b57ba5d43fedd7aa5e6c65c9c01089dc71edc4b4743416ad42800442b0f3"}
Oct 31 20:15:13.439410 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:13.439035    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5" event={"ID":"2ae1a099-1503-4ccf-a8a2-611bf5acec79","Type":"ContainerStarted","Data":"9057e951ed5b7f1f2eeae7ef5200b975764c071374cedb38bbf6b872ec388526"}
Oct 31 20:15:13.555758 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:13.555708    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5" podStartSLOduration=1.555694396 podStartE2EDuration="1.555694396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:13.553889018 +0000 UTC m=+3532.083423071" watchObservedRunningTime="2026-10-31 20:15:13.555694396 +0000 UTC m=+3532.085228448"
Oct 31 20:15:13.971587 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:13.971552    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 31 20:15:13.971805 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:13.971620    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 31 20:15:14.446748 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:14.446704    5006 generic.go:356] "Generic (PLEG): container finished" podID="2ae1a099-1503-4ccf-a8a2-611bf5acec79" containerID="db46b57ba5d43fedd7aa5e6c65c9c01089dc71edc4b4743416ad42800442b0f3" exitCode=0
Oct 31 20:15:14.447177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:14.446767    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5" event={"ID":"2ae1a099-1503-4ccf-a8a2-611bf5acec79","Type":"ContainerDied","Data":"db46b57ba5d43fedd7aa5e6c65c9c01089dc71edc4b4743416ad42800442b0f3"}
Oct 31 20:15:14.971030 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:14.970985    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 31 20:15:14.971222 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:14.971048    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 31 20:15:15.630269 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.630241    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:15.672430 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.672392    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-xks-ca\") pod \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") "
Oct 31 20:15:15.672624 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.672470    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae1a099-1503-4ccf-a8a2-611bf5acec79-kube-api-access-fbvqv\" (UniqueName: \"kubernetes.io/projected/2ae1a099-1503-4ccf-a8a2-611bf5acec79-kube-api-access-fbvqv\") pod \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") "
Oct 31 20:15:15.672624 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.672502    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-openshift-ca\") pod \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\" (UID: \"2ae1a099-1503-4ccf-a8a2-611bf5acec79\") "
Oct 31 20:15:15.672965 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.672939    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-openshift-ca" pod "2ae1a099-1503-4ccf-a8a2-611bf5acec79" (UID: "2ae1a099-1503-4ccf-a8a2-611bf5acec79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:15:15.674683 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.674662    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae1a099-1503-4ccf-a8a2-611bf5acec79-kube-api-access-fbvqv" pod "2ae1a099-1503-4ccf-a8a2-611bf5acec79" (UID: "2ae1a099-1503-4ccf-a8a2-611bf5acec79"). InnerVolumeSpecName "kube-api-access-fbvqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:15:15.674766 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.674743    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-xks-ca" pod "2ae1a099-1503-4ccf-a8a2-611bf5acec79" (UID: "2ae1a099-1503-4ccf-a8a2-611bf5acec79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:15:15.774221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.774134    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:15:15.774221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.774169    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbvqv\" (UniqueName: \"kubernetes.io/projected/2ae1a099-1503-4ccf-a8a2-611bf5acec79-kube-api-access-fbvqv\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:15:15.774221 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.774182    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae1a099-1503-4ccf-a8a2-611bf5acec79-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:15:15.972083 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.972042    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 31 20:15:15.972276 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:15.972099    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 31 20:15:16.459640 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:16.459605    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"
Oct 31 20:15:16.459640 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:16.459631    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5" event={"ID":"2ae1a099-1503-4ccf-a8a2-611bf5acec79","Type":"ContainerDied","Data":"9057e951ed5b7f1f2eeae7ef5200b975764c071374cedb38bbf6b872ec388526"}
Oct 31 20:15:16.459881 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:16.459696    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9057e951ed5b7f1f2eeae7ef5200b975764c071374cedb38bbf6b872ec388526"
Oct 31 20:15:16.971016 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:16.970976    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 31 20:15:16.971429 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:16.971033    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 31 20:15:17.971160 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:17.971117    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 31 20:15:17.971575 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:17.971178    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 31 20:15:19.971576 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:19.971523    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:15:19.972044 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:19.971599    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:15:20.046379 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:20.046332    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"]
Oct 31 20:15:20.080560 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:20.080521    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c343631f-hl6qj"]
Oct 31 20:15:20.410108 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:20.410041    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:20.676447 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:20.676344    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:21.013960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:21.013860    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:15:21.013960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:21.013929    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:15:21.597999 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:21.597955    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6baae907-f3cd-43dd-ba8a-1d9e6776b4de" path="/var/lib/kubelet/pods/6baae907-f3cd-43dd-ba8a-1d9e6776b4de/volumes"
Oct 31 20:15:21.734630 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:21.734590    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:21.735069 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:21.735047    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:21.735823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:21.735803    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k"
Oct 31 20:15:24.622693 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:24.622638    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:24.965203 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:24.965128    5006 scope.go:122] "RemoveContainer" containerID="c4417a2d99fa8f2e04bbaa34d40629a1ee623a2f73c44e60204aa1220f3bd4a6"
Oct 31 20:15:24.974028 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:24.974005    5006 scope.go:122] "RemoveContainer" containerID="d34d9476cbb3e4a9962b9006ff897e0b4f18971d3b30d23fef065be5dc145143"
Oct 31 20:15:27.019975 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:27.019939    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:31.696707 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:15:31.696625    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:35.393971 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:15:35.393929    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:16:05.395134 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:16:05.395090    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:16:06.047177 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:06.047137    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"]
Oct 31 20:16:06.097269 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:06.097234    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb219bea-4vc4c"]
Oct 31 20:16:07.597833 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:07.597791    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154c5972-4b1d-4d5d-9e70-fd2575cde1ca" path="/var/lib/kubelet/pods/154c5972-4b1d-4d5d-9e70-fd2575cde1ca/volumes"
Oct 31 20:16:14.340148 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:14.340110    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:16:22.280320 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:22.280290    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:16:22.365396 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:22.365361    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:16:25.009939 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:25.009902    5006 scope.go:122] "RemoveContainer" containerID="655aec9058995280a5043b8c5bc739722c28c99c798b1abd5123035e69809f9d"
Oct 31 20:16:25.019030 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:25.019009    5006 scope.go:122] "RemoveContainer" containerID="def9219d43af5e944ffd0f3e0a0363df20434a56afec29971a7bd2f12caae483"
Oct 31 20:16:32.043755 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:32.043716    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:16:35.042843 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:35.042805    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"]
Oct 31 20:16:35.093045 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:35.093013    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e242190-2l58k"]
Oct 31 20:16:35.393543 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:16:35.393501    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:16:35.598328 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:35.598292    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e292e4-cc4f-475f-8f95-8c3eb76fb5bc" path="/var/lib/kubelet/pods/79e292e4-cc4f-475f-8f95-8c3eb76fb5bc/volumes"
Oct 31 20:16:35.740388 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:35.740308    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:16:41.331730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.331694    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"]
Oct 31 20:16:41.332176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.332144    5006 cpu_manager.go:414] "containerMap: removing container" podUID="2ae1a099-1503-4ccf-a8a2-611bf5acec79" containerName="revoke-keys"
Oct 31 20:16:41.332176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.332158    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae1a099-1503-4ccf-a8a2-611bf5acec79" containerName="revoke-keys"
Oct 31 20:16:41.332271 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.332244    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae1a099-1503-4ccf-a8a2-611bf5acec79" containerName="revoke-keys"
Oct 31 20:16:41.336975 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.336954    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.399279 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.399247    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.399488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.399289    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.399488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.399314    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.399488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.399445    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.399488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.399480    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.399715 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.399505    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.399715 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.399525    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.399715 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.399574    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzhf\" (UniqueName: \"kubernetes.io/projected/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-kube-api-access-7dzhf\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.399715 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.399631    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.500518 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.500473    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.500756 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.500549    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.500756 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.500595    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.500756 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.500646    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.500756 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.500726    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzhf\" (UniqueName: \"kubernetes.io/projected/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-kube-api-access-7dzhf\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.501029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.500761    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.501029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.500826    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.501029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.500878    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.501029 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.500921    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.501774 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.501046    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.501774 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.501250    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.501774 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.501268    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.501774 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.501521    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.501774 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.501705    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.503385 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.503363    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.503493 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.503421    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.536564 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.536527    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"]
Oct 31 20:16:41.559094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.559053    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.559257 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.559114    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzhf\" (UniqueName: \"kubernetes.io/projected/1d98cc29-8ed6-4e88-a844-2a0863c5cfdf-kube-api-access-7dzhf\") pod \"maas-default-gateway-openshift-default-58d898bd55-t8qk4\" (UID: \"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:41.651247 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:41.651202    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:42.066408 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:16:42.066376    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d98cc29_8ed6_4e88_a844_2a0863c5cfdf.slice/crio-0bd4321f9c5d09229dd4ce33a83956fa49203ac2ca5f7b9589f9df6b727d7acd WatchSource:0}: Error finding container 0bd4321f9c5d09229dd4ce33a83956fa49203ac2ca5f7b9589f9df6b727d7acd: Status 404 returned error can't find the container with id 0bd4321f9c5d09229dd4ce33a83956fa49203ac2ca5f7b9589f9df6b727d7acd
Oct 31 20:16:42.068232 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:42.068200    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"]
Oct 31 20:16:42.069438 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:42.069401    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:16:42.069554 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:42.069466    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:16:42.069554 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:42.069496    5006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:16:42.950124 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:42.948855    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" event={"ID":"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf","Type":"ContainerStarted","Data":"eec249bae705c86b69124ad260a1ea8814dd56062331cf0b45841241200891a4"}
Oct 31 20:16:42.950843 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:42.950811    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" event={"ID":"1d98cc29-8ed6-4e88-a844-2a0863c5cfdf","Type":"ContainerStarted","Data":"0bd4321f9c5d09229dd4ce33a83956fa49203ac2ca5f7b9589f9df6b727d7acd"}
Oct 31 20:16:43.654673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:43.653811    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:43.654673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:43.653983    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-t8qk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:16:43.654673 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:43.654034    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" podUID="1d98cc29-8ed6-4e88-a844-2a0863c5cfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 31 20:16:44.087860 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:44.087826    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:16:44.652465 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:44.652428    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-t8qk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:16:44.652697 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:44.652486    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" podUID="1d98cc29-8ed6-4e88-a844-2a0863c5cfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 31 20:16:45.652056 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:45.652014    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-t8qk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:16:45.652696 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:45.652091    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" podUID="1d98cc29-8ed6-4e88-a844-2a0863c5cfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 31 20:16:46.652281 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:46.652244    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-t8qk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:16:46.652781 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:46.652297    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" podUID="1d98cc29-8ed6-4e88-a844-2a0863c5cfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 31 20:16:47.652475 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:47.652438    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-t8qk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:16:47.652930 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:47.652509    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" podUID="1d98cc29-8ed6-4e88-a844-2a0863c5cfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 31 20:16:47.753507 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:47.753435    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" podStartSLOduration=6.753414639 podStartE2EDuration="6.753414639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:16:43.128163206 +0000 UTC m=+3621.657697262" watchObservedRunningTime="2026-10-31 20:16:47.753414639 +0000 UTC m=+3626.282948757"
Oct 31 20:16:47.755312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:47.755284    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:16:48.652260 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:48.652221    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-t8qk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:16:48.652464 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:48.652288    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" podUID="1d98cc29-8ed6-4e88-a844-2a0863c5cfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 31 20:16:49.651906 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:49.651873    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-t8qk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:16:49.652345 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:49.651928    5006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" podUID="1d98cc29-8ed6-4e88-a844-2a0863c5cfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 31 20:16:50.656799 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:50.656768    5006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:50.657208 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:50.657156    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:16:50.657841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:16:50.657824    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4"
Oct 31 20:17:05.393537 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:17:05.393453    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:17:06.921904 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:17:06.921862    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:17:23.622367 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:17:23.622326    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:17:25.063776 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:17:25.063733    5006 scope.go:122] "RemoveContainer" containerID="dbf6b5a7a5295b6b1ebdf339785cd9ba407bda37f72820b86f5ea399c042d4ed"
Oct 31 20:17:25.084995 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:17:25.084965    5006 scope.go:122] "RemoveContainer" containerID="4d6a670629234602861d0f13fd0472851c4ffe9dd5b74484ab8140e7b678e6e7"
Oct 31 20:17:32.451692 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:17:32.451642    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:17:35.392989 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:17:35.392952    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:17:54.523218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:17:54.523172    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:18:05.143533 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:05.143489    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:18:05.393272 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:18:05.393231    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:18:16.736113 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:16.736073    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:18:19.047427 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:19.047389    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"]
Oct 31 20:18:19.097441 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:19.097404    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09cb56cf-5tkk7"]
Oct 31 20:18:19.597621 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:19.597585    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c751b999-02bd-4f36-814e-da70d5a2d94a" path="/var/lib/kubelet/pods/c751b999-02bd-4f36-814e-da70d5a2d94a/volumes"
Oct 31 20:18:20.047017 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:20.046917    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"]
Oct 31 20:18:20.097299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:20.097265    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5799008e-pw4cn"]
Oct 31 20:18:21.597533 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:21.597495    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75efbf1e-64e0-481d-b41c-1941975c5ff1" path="/var/lib/kubelet/pods/75efbf1e-64e0-481d-b41c-1941975c5ff1/volumes"
Oct 31 20:18:25.170568 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:25.170537    5006 scope.go:122] "RemoveContainer" containerID="ea5d8a9c38f2e320d27b54053cf0d884f395c2530b9bc33a77689d33720c67ab"
Oct 31 20:18:25.179823 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:25.179803    5006 scope.go:122] "RemoveContainer" containerID="f91d1212878c7fb23de0a57efc3cea92a7765b9cb0be1539746c7c00b119a8cb"
Oct 31 20:18:25.188238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:25.188215    5006 scope.go:122] "RemoveContainer" containerID="43b1a38107fefea3578d9935139a13c5dfabde7c6ddfdf2926e8ad59386006c7"
Oct 31 20:18:32.781728 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:32.781611    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:18:35.394706 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:18:35.394669    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:18:40.043015 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:40.042975    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:18:49.110424 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:18:49.110385    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:19:05.393107 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:19:05.393068    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:19:08.026877 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:08.026839    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:19:19.719526 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:19.719488    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:19:24.056146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:24.056108    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"]
Oct 31 20:19:24.106363 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:24.106308    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eaee8bf-xsv68"]
Oct 31 20:19:25.218980 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:25.218947    5006 scope.go:122] "RemoveContainer" containerID="59f182507180c76c2b9b45c3d7da8ddfd5403cccf6d36ea09dd4151fb5cebb5a"
Oct 31 20:19:25.228342 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:25.228318    5006 scope.go:122] "RemoveContainer" containerID="0e594dd278cdcaec536fde43af1a8fbbf2aba9c2df4359d854bf4889142f956d"
Oct 31 20:19:25.237051 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:25.237026    5006 scope.go:122] "RemoveContainer" containerID="56ad4b29a1ec345be886450828b71df5c3f89e4e846166210dac050826117f48"
Oct 31 20:19:25.245635 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:25.245612    5006 scope.go:122] "RemoveContainer" containerID="0cf0fe7b190c97d0644148e9cdff1a163d8518e68d1c30d91fe1df39a1b89aea"
Oct 31 20:19:25.254444 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:25.254422    5006 scope.go:122] "RemoveContainer" containerID="a53996e1824f8f9c626744c0c6e0e148ee6451dbe66f5cf1a2158dfddd03d68e"
Oct 31 20:19:25.597196 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:25.597165    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e6ff55-5c2a-42a1-808d-2d85cdd291ff" path="/var/lib/kubelet/pods/b2e6ff55-5c2a-42a1-808d-2d85cdd291ff/volumes"
Oct 31 20:19:33.074606 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:33.074536    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:35.394059 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:19:35.394022    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:19:38.407610 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:38.407565    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:19:47.305896 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:19:47.305855    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:20:05.240482 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:20:05.240399    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:20:05.396877 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:20:05.396838    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:20:16.047307 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:20:16.047264    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"]
Oct 31 20:20:16.097789 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:20:16.097746    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-td2q5"]
Oct 31 20:20:17.597279 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:20:17.597244    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae1a099-1503-4ccf-a8a2-611bf5acec79" path="/var/lib/kubelet/pods/2ae1a099-1503-4ccf-a8a2-611bf5acec79/volumes"
Oct 31 20:20:18.324672 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:20:18.324616    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:20:25.291958 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:20:25.291920    5006 scope.go:122] "RemoveContainer" containerID="93a02ed4a8f1464a9bd60baeab8ac514e2cde352fc5c58cbdc504d033ad1f4ed"
Oct 31 20:20:25.300990 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:20:25.300956    5006 scope.go:122] "RemoveContainer" containerID="024cf633a39e88bb6bb3a09596bc4fef876345518d76f43f62b2a6e1c3cd866a"
Oct 31 20:20:33.411663 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:20:33.411616    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:35.394281 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:20:35.394240    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:20:50.743332 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:20:50.743288    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:21:05.394128 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:21:05.394087    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:21:12.049348 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:12.049308    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:21:14.717841 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:14.717788    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cccdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:21:14.718509 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:14.718470    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podUID="b14112fb-b3f3-4ba3-8e12-af608d6c6beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:21:22.313593 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:22.313563    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:21:22.397473 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:22.397445    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:21:22.835995 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:22.835950    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:21:25.328392 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:25.328356    5006 scope.go:122] "RemoveContainer" containerID="db46b57ba5d43fedd7aa5e6c65c9c01089dc71edc4b4743416ad42800442b0f3"
Oct 31 20:21:33.758931 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:33.758818    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:35.394136 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:21:35.394097    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:21:44.806348 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:44.806306    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:45.482498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:45.482460    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:45.748448 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:45.748344    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:49.997981 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:49.997936    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:50.248237 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:50.248138    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:53.330497 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:53.330460    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:53.552008 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:53.551961    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:56.621480 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:56.621443    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:56.831436 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:21:56.831404    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:01.695877 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:01.695834    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:04.135940 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:04.135905    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:22:05.392869 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:22:05.392823    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:22:06.596420 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:06.596381    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:11.554294 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:11.554249    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:14.691073 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:14.691032    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:15.274259 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:15.274226    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:15.525714 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:15.525629    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:16.903416 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:16.903378    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:17.753787 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:17.753742    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:21.766953 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:21.766915    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:22.057513 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:22.057426    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:22:26.668349 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:26.668304    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:31.478084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:31.478036    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:34.151312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:34.151272    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:35.396562 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:22:35.396525    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:22:36.380087 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:36.380054    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:41.193065 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:41.193028    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:46.074062 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:46.074002    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:47.923603 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:47.923560    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:22:50.961068 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:50.961030    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:55.894421 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:22:55.894385    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:00.714501 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:00.714462    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:02.347041 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.347007    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"]
Oct 31 20:23:02.350639 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.350619    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:02.454468 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.454425    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:23:02.454468 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.454428    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:23:02.454812 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.454495    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:23:02.454812 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.454565    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:23:02.456476 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.456451    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"]
Oct 31 20:23:02.459971 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.459949    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae91ac2-hlxw2\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:02.460042 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.459996    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbkt8\" (UniqueName: \"kubernetes.io/projected/231785d9-b8bb-4769-8d6a-9c33372d8086-kube-api-access-sbkt8\") pod \"maas-api-revoke-sub-fae91ac2-hlxw2\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:02.460108 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.460082    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae91ac2-hlxw2\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:02.561520 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.561476    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae91ac2-hlxw2\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:02.561745 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.561597    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae91ac2-hlxw2\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:02.561745 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.561640    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbkt8\" (UniqueName: \"kubernetes.io/projected/231785d9-b8bb-4769-8d6a-9c33372d8086-kube-api-access-sbkt8\") pod \"maas-api-revoke-sub-fae91ac2-hlxw2\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:02.562271 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.562246    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae91ac2-hlxw2\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:02.563885 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.563861    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae91ac2-hlxw2\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:02.620071 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.619993    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbkt8\" (UniqueName: \"kubernetes.io/projected/231785d9-b8bb-4769-8d6a-9c33372d8086-kube-api-access-sbkt8\") pod \"maas-api-revoke-sub-fae91ac2-hlxw2\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:02.664009 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:02.663972    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:03.030607 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:23:03.030575    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231785d9_b8bb_4769_8d6a_9c33372d8086.slice/crio-f76cbd6b12b7523104f9b8dc1d91263ff31bdbe6ca8a225272f2a9c73df6877c WatchSource:0}: Error finding container f76cbd6b12b7523104f9b8dc1d91263ff31bdbe6ca8a225272f2a9c73df6877c: Status 404 returned error can't find the container with id f76cbd6b12b7523104f9b8dc1d91263ff31bdbe6ca8a225272f2a9c73df6877c
Oct 31 20:23:03.033180 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:03.033126    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"]
Oct 31 20:23:03.129485 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:03.129444    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2" event={"ID":"231785d9-b8bb-4769-8d6a-9c33372d8086","Type":"ContainerStarted","Data":"f76cbd6b12b7523104f9b8dc1d91263ff31bdbe6ca8a225272f2a9c73df6877c"}
Oct 31 20:23:04.136800 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:04.136760    5006 generic.go:356] "Generic (PLEG): container finished" podID="231785d9-b8bb-4769-8d6a-9c33372d8086" containerID="03a48f60895d07e1d5e97086d1a4084738470196c739487dbb1e22e62de44bf3" exitCode=0
Oct 31 20:23:04.137213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:04.136880    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2" event={"ID":"231785d9-b8bb-4769-8d6a-9c33372d8086","Type":"ContainerDied","Data":"03a48f60895d07e1d5e97086d1a4084738470196c739487dbb1e22e62de44bf3"}
Oct 31 20:23:05.289176 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.289149    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:05.393528 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:23:05.393444    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:23:05.393757 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.393736    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231785d9-b8bb-4769-8d6a-9c33372d8086-kube-api-access-sbkt8\" (UniqueName: \"kubernetes.io/projected/231785d9-b8bb-4769-8d6a-9c33372d8086-kube-api-access-sbkt8\") pod \"231785d9-b8bb-4769-8d6a-9c33372d8086\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") "
Oct 31 20:23:05.393833 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.393788    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-xks-ca\") pod \"231785d9-b8bb-4769-8d6a-9c33372d8086\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") "
Oct 31 20:23:05.393899 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.393853    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-openshift-ca\") pod \"231785d9-b8bb-4769-8d6a-9c33372d8086\" (UID: \"231785d9-b8bb-4769-8d6a-9c33372d8086\") "
Oct 31 20:23:05.394218 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.394195    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-openshift-ca" pod "231785d9-b8bb-4769-8d6a-9c33372d8086" (UID: "231785d9-b8bb-4769-8d6a-9c33372d8086"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:23:05.395881 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.395861    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-xks-ca" pod "231785d9-b8bb-4769-8d6a-9c33372d8086" (UID: "231785d9-b8bb-4769-8d6a-9c33372d8086"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:23:05.395960 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.395946    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231785d9-b8bb-4769-8d6a-9c33372d8086-kube-api-access-sbkt8" pod "231785d9-b8bb-4769-8d6a-9c33372d8086" (UID: "231785d9-b8bb-4769-8d6a-9c33372d8086"). InnerVolumeSpecName "kube-api-access-sbkt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:23:05.495282 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.495245    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:23:05.495282 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.495278    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbkt8\" (UniqueName: \"kubernetes.io/projected/231785d9-b8bb-4769-8d6a-9c33372d8086-kube-api-access-sbkt8\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:23:05.495282 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.495288    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231785d9-b8bb-4769-8d6a-9c33372d8086-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:23:05.582966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:05.582930    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:06.149572 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:06.149527    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2" event={"ID":"231785d9-b8bb-4769-8d6a-9c33372d8086","Type":"ContainerDied","Data":"f76cbd6b12b7523104f9b8dc1d91263ff31bdbe6ca8a225272f2a9c73df6877c"}
Oct 31 20:23:06.149572 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:06.149573    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"
Oct 31 20:23:06.149828 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:06.149586    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f76cbd6b12b7523104f9b8dc1d91263ff31bdbe6ca8a225272f2a9c73df6877c"
Oct 31 20:23:10.405680 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:10.405600    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:14.156067 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.156033    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:14.713981 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.713925    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"]
Oct 31 20:23:14.714467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.714448    5006 cpu_manager.go:414] "containerMap: removing container" podUID="231785d9-b8bb-4769-8d6a-9c33372d8086" containerName="revoke-keys"
Oct 31 20:23:14.714467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.714466    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231785d9-b8bb-4769-8d6a-9c33372d8086" containerName="revoke-keys"
Oct 31 20:23:14.714669 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.714542    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="231785d9-b8bb-4769-8d6a-9c33372d8086" containerName="revoke-keys"
Oct 31 20:23:14.718342 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.718317    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"]
Oct 31 20:23:14.718457 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.718409    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:14.771288 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.771250    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:23:14.771288 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.771261    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:23:14.771288 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.771287    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:23:14.771545 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.771296    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:23:14.784984 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.784954    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:14.785145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.785096    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:14.785191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.785152    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5cx9\" (UniqueName: \"kubernetes.io/projected/f9f5b59c-7800-4caa-913d-5122dc8b41c7-kube-api-access-q5cx9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:14.885847 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.885807    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:14.886055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.885874    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5cx9\" (UniqueName: \"kubernetes.io/projected/f9f5b59c-7800-4caa-913d-5122dc8b41c7-kube-api-access-q5cx9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:14.886055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.885907    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:14.886597 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.886570    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:14.888401 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.888373    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:14.943240 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:14.943210    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5cx9\" (UniqueName: \"kubernetes.io/projected/f9f5b59c-7800-4caa-913d-5122dc8b41c7-kube-api-access-q5cx9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:15.032062 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:15.031970    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:15.401049 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:23:15.401021    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f5b59c_7800_4caa_913d_5122dc8b41c7.slice/crio-6bd8fbede9835412513fc017f856da47498cce5995f9d89aa90a7bed64c5ab4f WatchSource:0}: Error finding container 6bd8fbede9835412513fc017f856da47498cce5995f9d89aa90a7bed64c5ab4f: Status 404 returned error can't find the container with id 6bd8fbede9835412513fc017f856da47498cce5995f9d89aa90a7bed64c5ab4f
Oct 31 20:23:15.402875 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:15.402736    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"]
Oct 31 20:23:15.564920 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:15.564886    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:16.202245 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:16.202214    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg" event={"ID":"f9f5b59c-7800-4caa-913d-5122dc8b41c7","Type":"ContainerStarted","Data":"0a0533a70a22ee0260d6d96244368bc669155f996386c8bc3f2b287d721ce7e5"}
Oct 31 20:23:16.202406 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:16.202271    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg" event={"ID":"f9f5b59c-7800-4caa-913d-5122dc8b41c7","Type":"ContainerStarted","Data":"6bd8fbede9835412513fc017f856da47498cce5995f9d89aa90a7bed64c5ab4f"}
Oct 31 20:23:16.317381 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:16.317330    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg" podStartSLOduration=2.317314419 podStartE2EDuration="2.317314419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:23:16.315498384 +0000 UTC m=+4014.845032431" watchObservedRunningTime="2026-10-31 20:23:16.317314419 +0000 UTC m=+4014.846848473"
Oct 31 20:23:17.040104 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:17.040057    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:23:17.211773 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:17.211725    5006 generic.go:356] "Generic (PLEG): container finished" podID="f9f5b59c-7800-4caa-913d-5122dc8b41c7" containerID="0a0533a70a22ee0260d6d96244368bc669155f996386c8bc3f2b287d721ce7e5" exitCode=0
Oct 31 20:23:17.212040 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:17.211886    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg" event={"ID":"f9f5b59c-7800-4caa-913d-5122dc8b41c7","Type":"ContainerDied","Data":"0a0533a70a22ee0260d6d96244368bc669155f996386c8bc3f2b287d721ce7e5"}
Oct 31 20:23:18.367669 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:18.367626    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:18.423488 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:18.423452    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-xks-ca\") pod \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") "
Oct 31 20:23:18.423709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:18.423508    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f5b59c-7800-4caa-913d-5122dc8b41c7-kube-api-access-q5cx9\" (UniqueName: \"kubernetes.io/projected/f9f5b59c-7800-4caa-913d-5122dc8b41c7-kube-api-access-q5cx9\") pod \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") "
Oct 31 20:23:18.423709 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:18.423604    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-openshift-ca\") pod \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\" (UID: \"f9f5b59c-7800-4caa-913d-5122dc8b41c7\") "
Oct 31 20:23:18.424184 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:18.424135    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-openshift-ca" pod "f9f5b59c-7800-4caa-913d-5122dc8b41c7" (UID: "f9f5b59c-7800-4caa-913d-5122dc8b41c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:23:18.425917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:18.425888    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f5b59c-7800-4caa-913d-5122dc8b41c7-kube-api-access-q5cx9" pod "f9f5b59c-7800-4caa-913d-5122dc8b41c7" (UID: "f9f5b59c-7800-4caa-913d-5122dc8b41c7"). InnerVolumeSpecName "kube-api-access-q5cx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:23:18.426007 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:18.425899    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-xks-ca" pod "f9f5b59c-7800-4caa-913d-5122dc8b41c7" (UID: "f9f5b59c-7800-4caa-913d-5122dc8b41c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:23:18.528731 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:18.528472    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:23:18.528731 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:18.528531    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5cx9\" (UniqueName: \"kubernetes.io/projected/f9f5b59c-7800-4caa-913d-5122dc8b41c7-kube-api-access-q5cx9\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:23:18.528731 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:18.528549    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9f5b59c-7800-4caa-913d-5122dc8b41c7-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:23:19.188141 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:19.188101    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:19.224255 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:19.224221    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"
Oct 31 20:23:19.224451 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:19.224252    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg" event={"ID":"f9f5b59c-7800-4caa-913d-5122dc8b41c7","Type":"ContainerDied","Data":"6bd8fbede9835412513fc017f856da47498cce5995f9d89aa90a7bed64c5ab4f"}
Oct 31 20:23:19.224451 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:19.224308    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd8fbede9835412513fc017f856da47498cce5995f9d89aa90a7bed64c5ab4f"
Oct 31 20:23:20.690299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:20.690251    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:25.564706 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:25.564636    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:26.589169 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:26.589122    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:26.840033 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:26.839948    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:30.362153 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:30.362122    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:34.466813 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:34.466775    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:34.681164 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:34.681129    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:35.406891 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:23:35.406846    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:23:36.185203 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:36.185168    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:36.460720 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:36.460613    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:36.705477 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:36.705439    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"]
Oct 31 20:23:36.755581 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:36.755494    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b50005-k88mg"]
Oct 31 20:23:37.183312 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:37.183279    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:37.597750 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:37.597710    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f5b59c-7800-4caa-913d-5122dc8b41c7" path="/var/lib/kubelet/pods/f9f5b59c-7800-4caa-913d-5122dc8b41c7/volumes"
Oct 31 20:23:39.004588 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:39.004553    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:23:42.900269 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:42.900231    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:43.222577 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:43.222489    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:43.439299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:43.439267    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:56.044039 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:56.044002    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:56.268525 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:23:56.268495    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:01.744469 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:24:01.744431    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:05.393008 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:24:05.392965    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:24:33.943922 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:24:33.943841    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:34.168902 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:24:34.168862    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:34.758309 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:24:34.758276    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:35.393061 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:24:35.393022    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:25:05.394939 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:25:05.394899    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:25:27.253069 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:25:27.253025    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:25:29.684875 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:25:29.684827    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cccdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:25:29.685406 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:25:29.684898    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podUID="b14112fb-b3f3-4ba3-8e12-af608d6c6beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:25:35.071085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:25:35.071022    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:35.394796 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:25:35.394753    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:25:44.533084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:25:44.533040    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:25:51.714804 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:25:51.714754    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-t8qk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:25:51.715435 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:25:51.715395    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" podUID="1d98cc29-8ed6-4e88-a844-2a0863c5cfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:25:53.352079 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:25:53.352041    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:26:05.393318 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:26:05.393231    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:26:13.112209 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:13.112174    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:13.338190 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:13.338146    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:26:13.750358 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:13.750317    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:13.971948 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:13.971919    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:17.402555 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:17.402518    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:17.638762 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:17.638715    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:20.712318 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:20.712270    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:20.951494 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:20.951456    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:22.345616 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:22.345588    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:26:22.428025 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:22.427996    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:26:24.015568 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:24.015530    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:24.229139 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:24.229106    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:24.453193 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:24.453163    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:29.096264 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:29.096226    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:31.544121 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:31.544085    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:26:34.100829 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:34.100780    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:35.394426 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:26:35.394384    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:26:35.413946 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:35.413912    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:38.973153 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:38.973116    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:42.040521 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:42.040475    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:42.660997 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:42.660952    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:42.894050 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:42.894011    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:44.305919 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:44.305883    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:46.389711 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:46.389644    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:49.173811 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:49.173775    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:50.631329 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:50.631286    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:26:54.157955 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:54.157917    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:59.009469 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:26:59.009370    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:03.870766 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:03.870715    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:05.394275 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:27:05.394228    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:27:08.797813 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:08.797778    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:13.787620 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:13.787578    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:15.449878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:15.449841    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:27:18.594848 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:18.594809    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:23.502906 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:23.502865    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:28.286956 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:28.286915    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:30.440295 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.440257    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"]
Oct 31 20:27:30.440958 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.440894    5006 cpu_manager.go:414] "containerMap: removing container" podUID="f9f5b59c-7800-4caa-913d-5122dc8b41c7" containerName="revoke-keys"
Oct 31 20:27:30.440958 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.440916    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f5b59c-7800-4caa-913d-5122dc8b41c7" containerName="revoke-keys"
Oct 31 20:27:30.441094 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.441057    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f5b59c-7800-4caa-913d-5122dc8b41c7" containerName="revoke-keys"
Oct 31 20:27:30.444217 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.444193    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:30.498084 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.498042    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7cace51-wgrrr\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:30.498296 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.498156    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwfm6\" (UniqueName: \"kubernetes.io/projected/1c5ac6f5-911d-4c28-bed2-94491b65510d-kube-api-access-dwfm6\") pod \"maas-api-revoke-sub-b7cace51-wgrrr\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:30.498296 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.498180    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7cace51-wgrrr\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:30.506124 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.506084    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"]
Oct 31 20:27:30.552202 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.552168    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:27:30.552202 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.552179    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:27:30.552439 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.552221    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:27:30.552439 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.552303    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:27:30.598889 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.598851    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7cace51-wgrrr\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:30.599100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.598912    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwfm6\" (UniqueName: \"kubernetes.io/projected/1c5ac6f5-911d-4c28-bed2-94491b65510d-kube-api-access-dwfm6\") pod \"maas-api-revoke-sub-b7cace51-wgrrr\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:30.599100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.598930    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7cace51-wgrrr\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:30.599684 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.599622    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7cace51-wgrrr\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:30.601588 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.601559    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7cace51-wgrrr\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:30.656602 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.656570    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwfm6\" (UniqueName: \"kubernetes.io/projected/1c5ac6f5-911d-4c28-bed2-94491b65510d-kube-api-access-dwfm6\") pod \"maas-api-revoke-sub-b7cace51-wgrrr\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:30.757545 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:30.757439    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:31.124162 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:27:31.124130    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5ac6f5_911d_4c28_bed2_94491b65510d.slice/crio-e1b3fe3d3e7ab90f7bb9bac38a15a5a3aa1fcbff6a7e20f9545aaf41e0a7e1c2 WatchSource:0}: Error finding container e1b3fe3d3e7ab90f7bb9bac38a15a5a3aa1fcbff6a7e20f9545aaf41e0a7e1c2: Status 404 returned error can't find the container with id e1b3fe3d3e7ab90f7bb9bac38a15a5a3aa1fcbff6a7e20f9545aaf41e0a7e1c2
Oct 31 20:27:31.126299 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:31.126274    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"]
Oct 31 20:27:31.503661 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:31.503612    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr" event={"ID":"1c5ac6f5-911d-4c28-bed2-94491b65510d","Type":"ContainerStarted","Data":"3eac80843542231ea468477c19e1ab3bbcd174d40365d36e81ef2f3e272717de"}
Oct 31 20:27:31.504154 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:31.503682    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr" event={"ID":"1c5ac6f5-911d-4c28-bed2-94491b65510d","Type":"ContainerStarted","Data":"e1b3fe3d3e7ab90f7bb9bac38a15a5a3aa1fcbff6a7e20f9545aaf41e0a7e1c2"}
Oct 31 20:27:31.620334 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:31.620273    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr" podStartSLOduration=1.620256481 podStartE2EDuration="1.620256481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:27:31.619363681 +0000 UTC m=+4270.148897738" watchObservedRunningTime="2026-10-31 20:27:31.620256481 +0000 UTC m=+4270.149790525"
Oct 31 20:27:32.510605 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:32.510568    5006 generic.go:356] "Generic (PLEG): container finished" podID="1c5ac6f5-911d-4c28-bed2-94491b65510d" containerID="3eac80843542231ea468477c19e1ab3bbcd174d40365d36e81ef2f3e272717de" exitCode=0
Oct 31 20:27:32.510605 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:32.510610    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr" event={"ID":"1c5ac6f5-911d-4c28-bed2-94491b65510d","Type":"ContainerDied","Data":"3eac80843542231ea468477c19e1ab3bbcd174d40365d36e81ef2f3e272717de"}
Oct 31 20:27:33.184874 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.184832    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:33.667262 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.667227    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:33.737214 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.737180    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-openshift-ca\") pod \"1c5ac6f5-911d-4c28-bed2-94491b65510d\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") "
Oct 31 20:27:33.737408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.737249    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5ac6f5-911d-4c28-bed2-94491b65510d-kube-api-access-dwfm6\" (UniqueName: \"kubernetes.io/projected/1c5ac6f5-911d-4c28-bed2-94491b65510d-kube-api-access-dwfm6\") pod \"1c5ac6f5-911d-4c28-bed2-94491b65510d\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") "
Oct 31 20:27:33.737408 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.737318    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-xks-ca\") pod \"1c5ac6f5-911d-4c28-bed2-94491b65510d\" (UID: \"1c5ac6f5-911d-4c28-bed2-94491b65510d\") "
Oct 31 20:27:33.737629 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.737607    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-openshift-ca" pod "1c5ac6f5-911d-4c28-bed2-94491b65510d" (UID: "1c5ac6f5-911d-4c28-bed2-94491b65510d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:27:33.739399 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.739376    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-xks-ca" pod "1c5ac6f5-911d-4c28-bed2-94491b65510d" (UID: "1c5ac6f5-911d-4c28-bed2-94491b65510d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:27:33.739489 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.739468    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5ac6f5-911d-4c28-bed2-94491b65510d-kube-api-access-dwfm6" pod "1c5ac6f5-911d-4c28-bed2-94491b65510d" (UID: "1c5ac6f5-911d-4c28-bed2-94491b65510d"). InnerVolumeSpecName "kube-api-access-dwfm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:27:33.838994 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.838949    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:27:33.839210 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.839007    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5ac6f5-911d-4c28-bed2-94491b65510d-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:27:33.839210 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:33.839020    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwfm6\" (UniqueName: \"kubernetes.io/projected/1c5ac6f5-911d-4c28-bed2-94491b65510d-kube-api-access-dwfm6\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:27:34.524537 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:34.524501    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"
Oct 31 20:27:34.524537 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:34.524513    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr" event={"ID":"1c5ac6f5-911d-4c28-bed2-94491b65510d","Type":"ContainerDied","Data":"e1b3fe3d3e7ab90f7bb9bac38a15a5a3aa1fcbff6a7e20f9545aaf41e0a7e1c2"}
Oct 31 20:27:34.524796 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:34.524600    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1b3fe3d3e7ab90f7bb9bac38a15a5a3aa1fcbff6a7e20f9545aaf41e0a7e1c2"
Oct 31 20:27:35.393711 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:27:35.393669    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:27:35.810332 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:35.810226    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:37.970638 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:37.970603    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:41.585472 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:41.585436    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:42.031060 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.031025    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"]
Oct 31 20:27:42.031454 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.031442    5006 cpu_manager.go:414] "containerMap: removing container" podUID="1c5ac6f5-911d-4c28-bed2-94491b65510d" containerName="revoke-keys"
Oct 31 20:27:42.031502 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.031455    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5ac6f5-911d-4c28-bed2-94491b65510d" containerName="revoke-keys"
Oct 31 20:27:42.031552 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.031542    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5ac6f5-911d-4c28-bed2-94491b65510d" containerName="revoke-keys"
Oct 31 20:27:42.035034 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.035017    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.087949 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.087917    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"]
Oct 31 20:27:42.117034 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.116991    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt6kk\" (UniqueName: \"kubernetes.io/projected/498731ec-5230-41a8-bd58-5ff6501f8284-kube-api-access-mt6kk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.117246 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.117116    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.117246 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.117178    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.136458 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.136421    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:27:42.136609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.136468    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:27:42.136609 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.136491    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:27:42.137724 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.137705    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:27:42.218386 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.218336    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt6kk\" (UniqueName: \"kubernetes.io/projected/498731ec-5230-41a8-bd58-5ff6501f8284-kube-api-access-mt6kk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.218586 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.218427    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.218586 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.218469    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.219198 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.219164    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.220977 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.220947    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.276216 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.276180    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt6kk\" (UniqueName: \"kubernetes.io/projected/498731ec-5230-41a8-bd58-5ff6501f8284-kube-api-access-mt6kk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.348435 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.348397    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:42.716211 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:27:42.716173    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498731ec_5230_41a8_bd58_5ff6501f8284.slice/crio-f7b596a23a58e0edd15d14d7a59bfbb6bbe78d3860ac18d84cee25b0495341db WatchSource:0}: Error finding container f7b596a23a58e0edd15d14d7a59bfbb6bbe78d3860ac18d84cee25b0495341db: Status 404 returned error can't find the container with id f7b596a23a58e0edd15d14d7a59bfbb6bbe78d3860ac18d84cee25b0495341db
Oct 31 20:27:42.718057 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.718035    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"]
Oct 31 20:27:42.948698 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:42.948662    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:43.245008 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:43.244967    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:27:43.573256 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:43.573221    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt" event={"ID":"498731ec-5230-41a8-bd58-5ff6501f8284","Type":"ContainerStarted","Data":"9a15c22f6ef28803fdb38a63c8fdca35152eff1806a9596020f78f78a64ef4d0"}
Oct 31 20:27:43.573498 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:43.573277    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt" event={"ID":"498731ec-5230-41a8-bd58-5ff6501f8284","Type":"ContainerStarted","Data":"f7b596a23a58e0edd15d14d7a59bfbb6bbe78d3860ac18d84cee25b0495341db"}
Oct 31 20:27:43.688725 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:43.688637    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt" podStartSLOduration=2.688621933 podStartE2EDuration="2.688621933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:27:43.686996324 +0000 UTC m=+4282.216530377" watchObservedRunningTime="2026-10-31 20:27:43.688621933 +0000 UTC m=+4282.218155983"
Oct 31 20:27:44.580478 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:44.580434    5006 generic.go:356] "Generic (PLEG): container finished" podID="498731ec-5230-41a8-bd58-5ff6501f8284" containerID="9a15c22f6ef28803fdb38a63c8fdca35152eff1806a9596020f78f78a64ef4d0" exitCode=0
Oct 31 20:27:44.580941 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:44.580519    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt" event={"ID":"498731ec-5230-41a8-bd58-5ff6501f8284","Type":"ContainerDied","Data":"9a15c22f6ef28803fdb38a63c8fdca35152eff1806a9596020f78f78a64ef4d0"}
Oct 31 20:27:45.727956 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:45.727927    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:45.854976 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:45.854944    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-openshift-ca\") pod \"498731ec-5230-41a8-bd58-5ff6501f8284\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") "
Oct 31 20:27:45.855213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:45.855031    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-xks-ca\") pod \"498731ec-5230-41a8-bd58-5ff6501f8284\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") "
Oct 31 20:27:45.855213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:45.855088    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/498731ec-5230-41a8-bd58-5ff6501f8284-kube-api-access-mt6kk\" (UniqueName: \"kubernetes.io/projected/498731ec-5230-41a8-bd58-5ff6501f8284-kube-api-access-mt6kk\") pod \"498731ec-5230-41a8-bd58-5ff6501f8284\" (UID: \"498731ec-5230-41a8-bd58-5ff6501f8284\") "
Oct 31 20:27:45.855410 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:45.855382    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-openshift-ca" pod "498731ec-5230-41a8-bd58-5ff6501f8284" (UID: "498731ec-5230-41a8-bd58-5ff6501f8284"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:27:45.857247 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:45.857213    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-xks-ca" pod "498731ec-5230-41a8-bd58-5ff6501f8284" (UID: "498731ec-5230-41a8-bd58-5ff6501f8284"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:27:45.857360 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:45.857263    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498731ec-5230-41a8-bd58-5ff6501f8284-kube-api-access-mt6kk" pod "498731ec-5230-41a8-bd58-5ff6501f8284" (UID: "498731ec-5230-41a8-bd58-5ff6501f8284"). InnerVolumeSpecName "kube-api-access-mt6kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:27:45.956173 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:45.956133    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt6kk\" (UniqueName: \"kubernetes.io/projected/498731ec-5230-41a8-bd58-5ff6501f8284-kube-api-access-mt6kk\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:27:45.956173 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:45.956166    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:27:45.956173 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:45.956176    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498731ec-5230-41a8-bd58-5ff6501f8284-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:27:46.591994 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:46.591964    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"
Oct 31 20:27:46.592186 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:46.591996    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt" event={"ID":"498731ec-5230-41a8-bd58-5ff6501f8284","Type":"ContainerDied","Data":"f7b596a23a58e0edd15d14d7a59bfbb6bbe78d3860ac18d84cee25b0495341db"}
Oct 31 20:27:46.592186 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:46.592042    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7b596a23a58e0edd15d14d7a59bfbb6bbe78d3860ac18d84cee25b0495341db"
Oct 31 20:27:46.614238 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:46.614201    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:48.073610 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:48.073566    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:52.967133 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:52.967060    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:53.964563 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:53.964516    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:54.177070 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:54.177033    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:57.771395 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:27:57.771360    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:02.025261 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:02.025222    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:03.641521 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:03.641484    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:03.934390 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:03.934309    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:04.175105 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:04.175065    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"]
Oct 31 20:28:04.225550 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:04.225468    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d28b6b-mkwbt"]
Oct 31 20:28:04.676101 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:04.676065    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:05.393197 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:28:05.393158    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:28:05.598229 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:05.598197    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="498731ec-5230-41a8-bd58-5ff6501f8284" path="/var/lib/kubelet/pods/498731ec-5230-41a8-bd58-5ff6501f8284/volumes"
Oct 31 20:28:06.053163 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:06.053130    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"]
Oct 31 20:28:06.103612 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:06.103572    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae91ac2-hlxw2"]
Oct 31 20:28:06.204698 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:06.204660    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:28:07.597309 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:07.597274    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231785d9-b8bb-4769-8d6a-9c33372d8086" path="/var/lib/kubelet/pods/231785d9-b8bb-4769-8d6a-9c33372d8086/volumes"
Oct 31 20:28:10.524688 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:10.524623    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:10.751423 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:10.751386    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:10.971393 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:10.971357    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:17.347586 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:17.347539    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:21.337410 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:21.337370    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:28:23.572531 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:23.572493    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:23.835599 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:23.835496    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:35.393670 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:28:35.393619    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:28:36.095161 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:36.095111    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:37.982439 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:37.982393    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:59.876802 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:28:59.876763    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:04.143428 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:04.143319    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:29:05.400423 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:29:05.400377    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:29:07.785870 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:07.785822    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:29:07.786393 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:07.785880    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:29:22.985440 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:22.985408    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:25.042819 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:25.042784    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:25.402427 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:25.402392    5006 scope.go:122] "RemoveContainer" containerID="0a0533a70a22ee0260d6d96244368bc669155f996386c8bc3f2b287d721ce7e5"
Oct 31 20:29:25.411574 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:25.411546    5006 scope.go:122] "RemoveContainer" containerID="03a48f60895d07e1d5e97086d1a4084738470196c739487dbb1e22e62de44bf3"
Oct 31 20:29:34.828789 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:34.828747    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:29:35.397420 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:29:35.397373    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:29:36.422594 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:36.422505    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:37.783821 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:37.783771    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:29:37.784335 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:37.783871    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:29:46.765729 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:46.765693    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:50.247700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:50.247641    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:29:51.364966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:51.364908    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:51.585811 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:51.585775    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:52.784867 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:52.784820    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vqn2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:29:52.785410 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:52.784908    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vqn2k" podUID="421392b6-59fb-4d6d-a35f-acfd09346096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:29:53.337880 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:29:53.337841    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:00.162675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.162087    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"]
Oct 31 20:30:00.163816 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.163791    5006 cpu_manager.go:414] "containerMap: removing container" podUID="498731ec-5230-41a8-bd58-5ff6501f8284" containerName="revoke-keys"
Oct 31 20:30:00.163974 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.163841    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="498731ec-5230-41a8-bd58-5ff6501f8284" containerName="revoke-keys"
Oct 31 20:30:00.164052 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.164026    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="498731ec-5230-41a8-bd58-5ff6501f8284" containerName="revoke-keys"
Oct 31 20:30:00.167228 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.167204    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"
Oct 31 20:30:00.171467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.171436    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6drpr\" (UniqueName: \"kubernetes.io/projected/77e4e933-ef5e-4525-921a-c0828388eab4-kube-api-access-6drpr\") pod \"maas-api-key-cleanup-29891310-kl955\" (UID: \"77e4e933-ef5e-4525-921a-c0828388eab4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"
Oct 31 20:30:00.171773 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.171750    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e4e933-ef5e-4525-921a-c0828388eab4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891310-kl955\" (UID: \"77e4e933-ef5e-4525-921a-c0828388eab4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"
Oct 31 20:30:00.236675 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.228134    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"]
Oct 31 20:30:00.273674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.273322    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e4e933-ef5e-4525-921a-c0828388eab4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891310-kl955\" (UID: \"77e4e933-ef5e-4525-921a-c0828388eab4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"
Oct 31 20:30:00.273674 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.273446    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6drpr\" (UniqueName: \"kubernetes.io/projected/77e4e933-ef5e-4525-921a-c0828388eab4-kube-api-access-6drpr\") pod \"maas-api-key-cleanup-29891310-kl955\" (UID: \"77e4e933-ef5e-4525-921a-c0828388eab4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"
Oct 31 20:30:00.279262 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.278332    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:30:00.279262 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.278435    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\""
Oct 31 20:30:00.279262 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.279188    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:30:00.285113 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.285081    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e4e933-ef5e-4525-921a-c0828388eab4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891310-kl955\" (UID: \"77e4e933-ef5e-4525-921a-c0828388eab4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"
Oct 31 20:30:00.369833 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.369797    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6drpr\" (UniqueName: \"kubernetes.io/projected/77e4e933-ef5e-4525-921a-c0828388eab4-kube-api-access-6drpr\") pod \"maas-api-key-cleanup-29891310-kl955\" (UID: \"77e4e933-ef5e-4525-921a-c0828388eab4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"
Oct 31 20:30:00.481759 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.481636    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"
Oct 31 20:30:00.986900 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:30:00.986861    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e4e933_ef5e_4525_921a_c0828388eab4.slice/crio-8d321ad70bd7a8b5ccbf54cc8f7614d386461958c6c32921e5109970cc8121a3 WatchSource:0}: Error finding container 8d321ad70bd7a8b5ccbf54cc8f7614d386461958c6c32921e5109970cc8121a3: Status 404 returned error can't find the container with id 8d321ad70bd7a8b5ccbf54cc8f7614d386461958c6c32921e5109970cc8121a3
Oct 31 20:30:00.989480 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:00.989454    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"]
Oct 31 20:30:01.457159 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:01.457111    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955" event={"ID":"77e4e933-ef5e-4525-921a-c0828388eab4","Type":"ContainerStarted","Data":"91f108b7c2c0e0d1ad9de839ce44e7cda7b1d214ceafc0e73e93e33323cfeb85"}
Oct 31 20:30:01.457659 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:01.457195    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955" event={"ID":"77e4e933-ef5e-4525-921a-c0828388eab4","Type":"ContainerStarted","Data":"8d321ad70bd7a8b5ccbf54cc8f7614d386461958c6c32921e5109970cc8121a3"}
Oct 31 20:30:01.572685 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:01.572552    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955" podStartSLOduration=1.572538459 podStartE2EDuration="1.572538459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:30:01.572045349 +0000 UTC m=+4420.101579402" watchObservedRunningTime="2026-10-31 20:30:01.572538459 +0000 UTC m=+4420.102072509"
Oct 31 20:30:02.463480 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:02.463446    5006 generic.go:356] "Generic (PLEG): container finished" podID="77e4e933-ef5e-4525-921a-c0828388eab4" containerID="91f108b7c2c0e0d1ad9de839ce44e7cda7b1d214ceafc0e73e93e33323cfeb85" exitCode=60
Oct 31 20:30:02.463934 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:02.463508    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955" event={"ID":"77e4e933-ef5e-4525-921a-c0828388eab4","Type":"ContainerDied","Data":"91f108b7c2c0e0d1ad9de839ce44e7cda7b1d214ceafc0e73e93e33323cfeb85"}
Oct 31 20:30:02.463934 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:02.463786    5006 scope.go:122] "RemoveContainer" containerID="91f108b7c2c0e0d1ad9de839ce44e7cda7b1d214ceafc0e73e93e33323cfeb85"
Oct 31 20:30:03.470350 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:03.470315    5006 generic.go:356] "Generic (PLEG): container finished" podID="77e4e933-ef5e-4525-921a-c0828388eab4" containerID="025a423ade3f8b9471c0d833d10f7f869bd39577667d8874ff9753439e8829c8" exitCode=60
Oct 31 20:30:03.470873 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:03.470402    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955" event={"ID":"77e4e933-ef5e-4525-921a-c0828388eab4","Type":"ContainerDied","Data":"025a423ade3f8b9471c0d833d10f7f869bd39577667d8874ff9753439e8829c8"}
Oct 31 20:30:03.470873 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:03.470489    5006 scope.go:122] "RemoveContainer" containerID="91f108b7c2c0e0d1ad9de839ce44e7cda7b1d214ceafc0e73e93e33323cfeb85"
Oct 31 20:30:03.470873 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:03.470664    5006 scope.go:122] "RemoveContainer" containerID="025a423ade3f8b9471c0d833d10f7f869bd39577667d8874ff9753439e8829c8"
Oct 31 20:30:03.471065 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:30:03.470932    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891310-kl955_odh-ai-gateway-infra(77e4e933-ef5e-4525-921a-c0828388eab4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955" podUID="77e4e933-ef5e-4525-921a-c0828388eab4"
Oct 31 20:30:04.309351 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:04.309313    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:30:04.476706 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:04.476636    5006 scope.go:122] "RemoveContainer" containerID="025a423ade3f8b9471c0d833d10f7f869bd39577667d8874ff9753439e8829c8"
Oct 31 20:30:04.477185 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:30:04.476922    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891310-kl955_odh-ai-gateway-infra(77e4e933-ef5e-4525-921a-c0828388eab4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955" podUID="77e4e933-ef5e-4525-921a-c0828388eab4"
Oct 31 20:30:05.392261 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:30:05.392219    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:30:15.031267 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:15.031220    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:30:15.591372 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:15.591325    5006 scope.go:122] "RemoveContainer" containerID="025a423ade3f8b9471c0d833d10f7f869bd39577667d8874ff9753439e8829c8"
Oct 31 20:30:16.581780 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:16.581728    5006 generic.go:356] "Generic (PLEG): container finished" podID="77e4e933-ef5e-4525-921a-c0828388eab4" containerID="79dc729da3e7981250c0de5117ac3ef63ee63c4b37431f89ead9e03e1e50f568" exitCode=60
Oct 31 20:30:16.581780 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:16.581768    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955" event={"ID":"77e4e933-ef5e-4525-921a-c0828388eab4","Type":"ContainerDied","Data":"79dc729da3e7981250c0de5117ac3ef63ee63c4b37431f89ead9e03e1e50f568"}
Oct 31 20:30:16.582346 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:16.581874    5006 scope.go:122] "RemoveContainer" containerID="025a423ade3f8b9471c0d833d10f7f869bd39577667d8874ff9753439e8829c8"
Oct 31 20:30:16.582407 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:16.582374    5006 scope.go:122] "RemoveContainer" containerID="79dc729da3e7981250c0de5117ac3ef63ee63c4b37431f89ead9e03e1e50f568"
Oct 31 20:30:16.582734 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:30:16.582692    5006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891310-kl955_odh-ai-gateway-infra(77e4e933-ef5e-4525-921a-c0828388eab4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955" podUID="77e4e933-ef5e-4525-921a-c0828388eab4"
Oct 31 20:30:17.755765 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:17.755716    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"]
Oct 31 20:30:18.061422 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.061381    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"
Oct 31 20:30:18.083296 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.083259    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e4e933-ef5e-4525-921a-c0828388eab4-kube-api-access-6drpr\" (UniqueName: \"kubernetes.io/projected/77e4e933-ef5e-4525-921a-c0828388eab4-kube-api-access-6drpr\") pod \"77e4e933-ef5e-4525-921a-c0828388eab4\" (UID: \"77e4e933-ef5e-4525-921a-c0828388eab4\") "
Oct 31 20:30:18.083494 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.083467    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e4e933-ef5e-4525-921a-c0828388eab4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e4e933-ef5e-4525-921a-c0828388eab4-maas-api-service-ca\") pod \"77e4e933-ef5e-4525-921a-c0828388eab4\" (UID: \"77e4e933-ef5e-4525-921a-c0828388eab4\") "
Oct 31 20:30:18.083925 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.083897    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e4e933-ef5e-4525-921a-c0828388eab4-maas-api-service-ca" pod "77e4e933-ef5e-4525-921a-c0828388eab4" (UID: "77e4e933-ef5e-4525-921a-c0828388eab4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:30:18.085816 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.085793    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e4e933-ef5e-4525-921a-c0828388eab4-kube-api-access-6drpr" pod "77e4e933-ef5e-4525-921a-c0828388eab4" (UID: "77e4e933-ef5e-4525-921a-c0828388eab4"). InnerVolumeSpecName "kube-api-access-6drpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:30:18.184933 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.184891    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6drpr\" (UniqueName: \"kubernetes.io/projected/77e4e933-ef5e-4525-921a-c0828388eab4-kube-api-access-6drpr\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:30:18.184933 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.184930    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e4e933-ef5e-4525-921a-c0828388eab4-maas-api-service-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:30:18.594240 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.594209    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"
Oct 31 20:30:18.594453 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.594243    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955" event={"ID":"77e4e933-ef5e-4525-921a-c0828388eab4","Type":"ContainerDied","Data":"8d321ad70bd7a8b5ccbf54cc8f7614d386461958c6c32921e5109970cc8121a3"}
Oct 31 20:30:18.594453 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.594299    5006 scope.go:122] "RemoveContainer" containerID="79dc729da3e7981250c0de5117ac3ef63ee63c4b37431f89ead9e03e1e50f568"
Oct 31 20:30:18.758594 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.758557    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"]
Oct 31 20:30:18.760959 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:18.760933    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-kl955"]
Oct 31 20:30:19.599878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:19.598070    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e4e933-ef5e-4525-921a-c0828388eab4" path="/var/lib/kubelet/pods/77e4e933-ef5e-4525-921a-c0828388eab4/volumes"
Oct 31 20:30:35.393465 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:30:35.393376    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:30:36.196395 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:36.196354    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:30:36.749571 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:36.749529    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:43.067368 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:43.067334    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:43.636729 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:43.636692    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:43.909842 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:43.909756    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:47.461702 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:47.461641    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:47.687061 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:47.687028    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:50.709599 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:50.709553    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:50.986435 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:50.986344    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:52.235438 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:52.235401    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:52.485564 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:52.485467    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:57.609711 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:57.609673    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:59.928895 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:30:59.928859    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:31:02.504833 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:02.504798    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:05.392906 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:31:05.392865    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:31:07.372834 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:07.372796    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:10.872896 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:10.872865    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:11.126045 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:11.125957    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:12.703347 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:12.703304    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:17.989444 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:17.989393    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:22.377593 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:22.377564    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:31:22.457901 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:22.457869    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:31:22.901126 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:22.901088    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:23.737741 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.737708    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"]
Oct 31 20:31:23.738284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.738179    5006 cpu_manager.go:414] "containerMap: removing container" podUID="77e4e933-ef5e-4525-921a-c0828388eab4" containerName="cleanup"
Oct 31 20:31:23.738284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.738192    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e4e933-ef5e-4525-921a-c0828388eab4" containerName="cleanup"
Oct 31 20:31:23.738284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.738209    5006 cpu_manager.go:414] "containerMap: removing container" podUID="77e4e933-ef5e-4525-921a-c0828388eab4" containerName="cleanup"
Oct 31 20:31:23.738284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.738214    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e4e933-ef5e-4525-921a-c0828388eab4" containerName="cleanup"
Oct 31 20:31:23.738284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.738228    5006 cpu_manager.go:414] "containerMap: removing container" podUID="77e4e933-ef5e-4525-921a-c0828388eab4" containerName="cleanup"
Oct 31 20:31:23.738284 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.738233    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e4e933-ef5e-4525-921a-c0828388eab4" containerName="cleanup"
Oct 31 20:31:23.738486 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.738303    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e4e933-ef5e-4525-921a-c0828388eab4" containerName="cleanup"
Oct 31 20:31:23.738486 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.738313    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e4e933-ef5e-4525-921a-c0828388eab4" containerName="cleanup"
Oct 31 20:31:23.738486 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.738324    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e4e933-ef5e-4525-921a-c0828388eab4" containerName="cleanup"
Oct 31 20:31:23.741458 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.741440    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:23.836848 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.836777    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:31:23.836848 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.836802    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:31:23.836848 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.836777    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:31:23.838965 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.838935    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"]
Oct 31 20:31:23.849285 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.849241    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:23.849485 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.849386    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:23.849485 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.849428    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpxll\" (UniqueName: \"kubernetes.io/projected/6c16d9fc-63d9-4b43-a732-dd9409549381-kube-api-access-hpxll\") pod \"maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:23.883778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.883744    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpkrx\""
Oct 31 20:31:23.950124 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.950084    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:23.950317 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.950157    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:23.950317 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.950177    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpxll\" (UniqueName: \"kubernetes.io/projected/6c16d9fc-63d9-4b43-a732-dd9409549381-kube-api-access-hpxll\") pod \"maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:23.950866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.950844    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:23.952463 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:23.952442    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:24.007499 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:24.007429    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpxll\" (UniqueName: \"kubernetes.io/projected/6c16d9fc-63d9-4b43-a732-dd9409549381-kube-api-access-hpxll\") pod \"maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:24.054449 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:24.054409    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:24.421631 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:31:24.421595    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c16d9fc_63d9_4b43_a732_dd9409549381.slice/crio-a3a0097a97e539470b00d4703f5915ec31c68275406d975549f697bba5c44f58 WatchSource:0}: Error finding container a3a0097a97e539470b00d4703f5915ec31c68275406d975549f697bba5c44f58: Status 404 returned error can't find the container with id a3a0097a97e539470b00d4703f5915ec31c68275406d975549f697bba5c44f58
Oct 31 20:31:24.423343 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:24.423310    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"]
Oct 31 20:31:24.951220 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:24.951182    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk" event={"ID":"6c16d9fc-63d9-4b43-a732-dd9409549381","Type":"ContainerStarted","Data":"e2de97fa97457b8b036438baabd1df044be3a5c47114e11a9d04a80c2f6e94f9"}
Oct 31 20:31:24.951727 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:24.951236    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk" event={"ID":"6c16d9fc-63d9-4b43-a732-dd9409549381","Type":"ContainerStarted","Data":"a3a0097a97e539470b00d4703f5915ec31c68275406d975549f697bba5c44f58"}
Oct 31 20:31:25.067113 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:25.067000    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk" podStartSLOduration=2.066986403 podStartE2EDuration="2.066986403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:31:25.06652075 +0000 UTC m=+4503.596054803" watchObservedRunningTime="2026-10-31 20:31:25.066986403 +0000 UTC m=+4503.596520453"
Oct 31 20:31:25.957766 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:25.957723    5006 generic.go:356] "Generic (PLEG): container finished" podID="6c16d9fc-63d9-4b43-a732-dd9409549381" containerID="e2de97fa97457b8b036438baabd1df044be3a5c47114e11a9d04a80c2f6e94f9" exitCode=0
Oct 31 20:31:25.958204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:25.957778    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk" event={"ID":"6c16d9fc-63d9-4b43-a732-dd9409549381","Type":"ContainerDied","Data":"e2de97fa97457b8b036438baabd1df044be3a5c47114e11a9d04a80c2f6e94f9"}
Oct 31 20:31:27.114452 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.114423    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:27.283243 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.283158    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c16d9fc-63d9-4b43-a732-dd9409549381-kube-api-access-hpxll\" (UniqueName: \"kubernetes.io/projected/6c16d9fc-63d9-4b43-a732-dd9409549381-kube-api-access-hpxll\") pod \"6c16d9fc-63d9-4b43-a732-dd9409549381\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") "
Oct 31 20:31:27.283401 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.283252    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-xks-ca\") pod \"6c16d9fc-63d9-4b43-a732-dd9409549381\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") "
Oct 31 20:31:27.283401 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.283294    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-openshift-ca\") pod \"6c16d9fc-63d9-4b43-a732-dd9409549381\" (UID: \"6c16d9fc-63d9-4b43-a732-dd9409549381\") "
Oct 31 20:31:27.283748 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.283711    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-openshift-ca" pod "6c16d9fc-63d9-4b43-a732-dd9409549381" (UID: "6c16d9fc-63d9-4b43-a732-dd9409549381"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:31:27.285423 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.285399    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c16d9fc-63d9-4b43-a732-dd9409549381-kube-api-access-hpxll" pod "6c16d9fc-63d9-4b43-a732-dd9409549381" (UID: "6c16d9fc-63d9-4b43-a732-dd9409549381"). InnerVolumeSpecName "kube-api-access-hpxll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:31:27.285521 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.285475    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-xks-ca" pod "6c16d9fc-63d9-4b43-a732-dd9409549381" (UID: "6c16d9fc-63d9-4b43-a732-dd9409549381"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:31:27.384939 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.384894    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpxll\" (UniqueName: \"kubernetes.io/projected/6c16d9fc-63d9-4b43-a732-dd9409549381-kube-api-access-hpxll\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:31:27.384939 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.384928    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:31:27.384939 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.384938    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c16d9fc-63d9-4b43-a732-dd9409549381-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:31:27.766840 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.766808    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:27.970051 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.969964    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk" event={"ID":"6c16d9fc-63d9-4b43-a732-dd9409549381","Type":"ContainerDied","Data":"a3a0097a97e539470b00d4703f5915ec31c68275406d975549f697bba5c44f58"}
Oct 31 20:31:27.970051 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.970013    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"
Oct 31 20:31:27.970051 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:27.970027    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a0097a97e539470b00d4703f5915ec31c68275406d975549f697bba5c44f58"
Oct 31 20:31:32.685956 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:32.685912    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:35.393014 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:31:35.392973    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:31:35.688459 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:35.688370    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:35.910839 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:35.910800    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:37.068565 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:37.068526    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:37.603268 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:37.603236    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:41.887945 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:41.887898    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:43.706973 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:43.706934    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:43.865085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:43.865049    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"]
Oct 31 20:31:43.915508 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:43.915478    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-47884b3f-gbqtk"]
Oct 31 20:31:44.744180 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:44.744133    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:45.596915 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:45.596868    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c16d9fc-63d9-4b43-a732-dd9409549381" path="/var/lib/kubelet/pods/6c16d9fc-63d9-4b43-a732-dd9409549381/volumes"
Oct 31 20:31:45.896075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:45.895990    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:46.608181 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:46.608137    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:46.827369 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:46.827331    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:47.640774 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:47.640737    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:31:50.122173 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:50.122139    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:50.385014 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:50.384910    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:53.436137 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:53.436083    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:53.665815 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:53.665780    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:53.944361 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:53.944319    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:54.201917 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:54.201836    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:54.438322 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:54.438284    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:59.119396 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:31:59.119357    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:01.533251 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:01.533152    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:32:04.089208 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:04.089102    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:05.393134 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:32:05.393096    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:32:08.989903 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:08.989816    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:09.754336 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:09.754299    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:14.174878 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:14.174839    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:18.980808 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:18.980773    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:23.873394 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:23.873355    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:28.769904 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:28.769870    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:33.675347 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:33.675312    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:34.047053 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:34.046972    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"]
Oct 31 20:32:34.097315 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:34.097280    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7cace51-wgrrr"]
Oct 31 20:32:35.393605 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:32:35.393563    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:32:35.602014 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:35.601977    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5ac6f5-911d-4c28-bed2-94491b65510d" path="/var/lib/kubelet/pods/1c5ac6f5-911d-4c28-bed2-94491b65510d/volumes"
Oct 31 20:32:37.460507 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:37.460471    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:38.567370 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:38.567332    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:43.477295 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:43.477258    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:48.284205 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:48.284168    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:53.169732 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:53.169691    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:32:58.060918 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:32:58.060884    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:03.016774 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:03.016734    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:05.394927 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:33:05.394876    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:33:07.769736 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:07.769688    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:12.766018 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:12.765974    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:17.574277 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:17.574233    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:22.663278 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:22.663238    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:27.414568 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:27.414528    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:32.280305 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.280214    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:32.757355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.757319    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"]
Oct 31 20:33:32.757866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.757850    5006 cpu_manager.go:414] "containerMap: removing container" podUID="6c16d9fc-63d9-4b43-a732-dd9409549381" containerName="revoke-keys"
Oct 31 20:33:32.757993 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.757868    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c16d9fc-63d9-4b43-a732-dd9409549381" containerName="revoke-keys"
Oct 31 20:33:32.757993 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.757933    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c16d9fc-63d9-4b43-a732-dd9409549381" containerName="revoke-keys"
Oct 31 20:33:32.761069 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.761047    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:32.771078 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.771054    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c545k\" (UniqueName: \"kubernetes.io/projected/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-kube-api-access-c545k\") pod \"maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:32.771190 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.771104    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:32.771243 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.771221    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:32.815400 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.815361    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"]
Oct 31 20:33:32.851479 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.851448    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:33:32.851479 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.851472    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:33:32.851715 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.851497    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:33:32.863370 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.863339    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpkrx\""
Oct 31 20:33:32.872319 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.872280    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:32.872494 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.872380    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c545k\" (UniqueName: \"kubernetes.io/projected/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-kube-api-access-c545k\") pod \"maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:32.872494 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.872433    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:32.873046 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.873026    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:32.874905 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.874886    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:32.937187 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:32.937155    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c545k\" (UniqueName: \"kubernetes.io/projected/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-kube-api-access-c545k\") pod \"maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:33.074201 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:33.074098    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:33.445338 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:33:33.445298    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776a7b56_79f8_4b48_a2b4_55467bbc2b2e.slice/crio-b24ebc84308f5183f84f97a3f47d4d630133c8831a95c220be001d9d54b23593 WatchSource:0}: Error finding container b24ebc84308f5183f84f97a3f47d4d630133c8831a95c220be001d9d54b23593: Status 404 returned error can't find the container with id b24ebc84308f5183f84f97a3f47d4d630133c8831a95c220be001d9d54b23593
Oct 31 20:33:33.447388 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:33.447321    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"]
Oct 31 20:33:33.608594 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:33.608562    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2" event={"ID":"776a7b56-79f8-4b48-a2b4-55467bbc2b2e","Type":"ContainerStarted","Data":"b24ebc84308f5183f84f97a3f47d4d630133c8831a95c220be001d9d54b23593"}
Oct 31 20:33:34.615827 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:34.615792    5006 generic.go:356] "Generic (PLEG): container finished" podID="776a7b56-79f8-4b48-a2b4-55467bbc2b2e" containerID="fc7a60886e625592baed44d82f528c7a33c5b7450666c4690204273aaa8a7cd3" exitCode=0
Oct 31 20:33:34.616236 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:34.615876    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2" event={"ID":"776a7b56-79f8-4b48-a2b4-55467bbc2b2e","Type":"ContainerDied","Data":"fc7a60886e625592baed44d82f528c7a33c5b7450666c4690204273aaa8a7cd3"}
Oct 31 20:33:35.395075 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:33:35.395038    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:33:35.784345 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:35.784318    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:35.904334 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:35.904296    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-xks-ca\") pod \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") "
Oct 31 20:33:35.904523 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:35.904370    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-kube-api-access-c545k\" (UniqueName: \"kubernetes.io/projected/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-kube-api-access-c545k\") pod \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") "
Oct 31 20:33:35.904523 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:35.904446    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-openshift-ca\") pod \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\" (UID: \"776a7b56-79f8-4b48-a2b4-55467bbc2b2e\") "
Oct 31 20:33:35.904955 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:35.904920    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-openshift-ca" pod "776a7b56-79f8-4b48-a2b4-55467bbc2b2e" (UID: "776a7b56-79f8-4b48-a2b4-55467bbc2b2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:33:35.906736 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:35.906702    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-kube-api-access-c545k" pod "776a7b56-79f8-4b48-a2b4-55467bbc2b2e" (UID: "776a7b56-79f8-4b48-a2b4-55467bbc2b2e"). InnerVolumeSpecName "kube-api-access-c545k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:33:35.906849 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:35.906749    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-xks-ca" pod "776a7b56-79f8-4b48-a2b4-55467bbc2b2e" (UID: "776a7b56-79f8-4b48-a2b4-55467bbc2b2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:33:36.005846 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:36.005752    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:33:36.005846 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:36.005786    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c545k\" (UniqueName: \"kubernetes.io/projected/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-kube-api-access-c545k\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:33:36.005846 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:36.005798    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776a7b56-79f8-4b48-a2b4-55467bbc2b2e-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:33:36.627838 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:36.627806    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"
Oct 31 20:33:36.628140 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:36.627840    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2" event={"ID":"776a7b56-79f8-4b48-a2b4-55467bbc2b2e","Type":"ContainerDied","Data":"b24ebc84308f5183f84f97a3f47d4d630133c8831a95c220be001d9d54b23593"}
Oct 31 20:33:36.628140 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:36.627893    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b24ebc84308f5183f84f97a3f47d4d630133c8831a95c220be001d9d54b23593"
Oct 31 20:33:37.501600 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:37.501521    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:37.735721 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:37.735688    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:38.019593 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:38.019543    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:42.594738 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:42.594703    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:44.738290 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:44.738258    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:44.953067 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:44.953030    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:47.369444 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:47.369399    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:56.668062 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:56.668022    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:58.205215 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:58.205177    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:58.484375 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:58.484289    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:58.725055 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:58.725023    5006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"]
Oct 31 20:33:58.775213 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:58.775168    5006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-12dd5c-cfqz2"]
Oct 31 20:33:59.193771 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:59.193697    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:33:59.597075 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:33:59.597035    5006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776a7b56-79f8-4b48-a2b4-55467bbc2b2e" path="/var/lib/kubelet/pods/776a7b56-79f8-4b48-a2b4-55467bbc2b2e/volumes"
Oct 31 20:34:00.825253 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:00.825209    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:34:05.057201 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:05.057157    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:34:05.281183 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:05.281151    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:34:05.393036 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:34:05.392993    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:34:05.535496 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:05.535452    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:34:11.928267 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:11.928227    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:34:14.683849 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:14.683795    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cccdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:34:14.684289 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:14.683878    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cccdj" podUID="b14112fb-b3f3-4ba3-8e12-af608d6c6beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:34:24.138019 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:24.137980    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:34:25.475469 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:25.475437    5006 scope.go:122] "RemoveContainer" containerID="9a15c22f6ef28803fdb38a63c8fdca35152eff1806a9596020f78f78a64ef4d0"
Oct 31 20:34:25.485039 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:25.485009    5006 scope.go:122] "RemoveContainer" containerID="3eac80843542231ea468477c19e1ab3bbcd174d40365d36e81ef2f3e272717de"
Oct 31 20:34:31.294359 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:31.294327    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:34:31.518975 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:31.518944    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:34:32.396794 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.396757    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"]
Oct 31 20:34:32.397249 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.397235    5006 cpu_manager.go:414] "containerMap: removing container" podUID="776a7b56-79f8-4b48-a2b4-55467bbc2b2e" containerName="revoke-keys"
Oct 31 20:34:32.397297 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.397250    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776a7b56-79f8-4b48-a2b4-55467bbc2b2e" containerName="revoke-keys"
Oct 31 20:34:32.397346 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.397326    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="776a7b56-79f8-4b48-a2b4-55467bbc2b2e" containerName="revoke-keys"
Oct 31 20:34:32.400586 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.400554    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:32.454945 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.454909    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"]
Oct 31 20:34:32.465988 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.465952    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-m2lcm\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:32.466197 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.466017    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwmfn\" (UniqueName: \"kubernetes.io/projected/540ee8ab-a2cd-484c-b1e5-23661ea5b251-kube-api-access-qwmfn\") pod \"maas-api-revoke-sub-f03629db-m2lcm\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:32.466197 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.466139    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-m2lcm\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:32.502536 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.502504    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:34:32.502760 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.502547    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:34:32.503937 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.503911    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:34:32.504078 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.503948    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpkrx\""
Oct 31 20:34:32.567448 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.567411    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-m2lcm\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:32.567635 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.567501    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-m2lcm\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:32.567635 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.567529    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwmfn\" (UniqueName: \"kubernetes.io/projected/540ee8ab-a2cd-484c-b1e5-23661ea5b251-kube-api-access-qwmfn\") pod \"maas-api-revoke-sub-f03629db-m2lcm\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:32.568152 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.568132    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-m2lcm\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:32.569931 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.569912    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-m2lcm\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:32.625039 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.625007    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwmfn\" (UniqueName: \"kubernetes.io/projected/540ee8ab-a2cd-484c-b1e5-23661ea5b251-kube-api-access-qwmfn\") pod \"maas-api-revoke-sub-f03629db-m2lcm\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:32.715144 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:32.715057    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:33.080782 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:34:33.080750    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540ee8ab_a2cd_484c_b1e5_23661ea5b251.slice/crio-79a44a62ad6308f4e5c41e9e868c3f70cbf3412d31f4b303b4eea1eb37fc7aed WatchSource:0}: Error finding container 79a44a62ad6308f4e5c41e9e868c3f70cbf3412d31f4b303b4eea1eb37fc7aed: Status 404 returned error can't find the container with id 79a44a62ad6308f4e5c41e9e868c3f70cbf3412d31f4b303b4eea1eb37fc7aed
Oct 31 20:34:33.082376 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:33.082354    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"]
Oct 31 20:34:33.930085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:33.930039    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm" event={"ID":"540ee8ab-a2cd-484c-b1e5-23661ea5b251","Type":"ContainerStarted","Data":"24352e8ae185be129cdc30ce37aaab37e93b91572cd4eb813da73a6df7cd5e44"}
Oct 31 20:34:33.930487 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:33.930106    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm" event={"ID":"540ee8ab-a2cd-484c-b1e5-23661ea5b251","Type":"ContainerStarted","Data":"79a44a62ad6308f4e5c41e9e868c3f70cbf3412d31f4b303b4eea1eb37fc7aed"}
Oct 31 20:34:34.045247 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:34.045174    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm" podStartSLOduration=2.04515977 podStartE2EDuration="2.04515977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:34:34.044049901 +0000 UTC m=+4692.573583953" watchObservedRunningTime="2026-10-31 20:34:34.04515977 +0000 UTC m=+4692.574693850"
Oct 31 20:34:34.937441 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:34.937405    5006 generic.go:356] "Generic (PLEG): container finished" podID="540ee8ab-a2cd-484c-b1e5-23661ea5b251" containerID="24352e8ae185be129cdc30ce37aaab37e93b91572cd4eb813da73a6df7cd5e44" exitCode=0
Oct 31 20:34:34.937892 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:34.937495    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm" event={"ID":"540ee8ab-a2cd-484c-b1e5-23661ea5b251","Type":"ContainerDied","Data":"24352e8ae185be129cdc30ce37aaab37e93b91572cd4eb813da73a6df7cd5e44"}
Oct 31 20:34:35.393870 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:34:35.393834    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:34:36.084557 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.084526    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:36.204588 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.204500    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-openshift-ca\") pod \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") "
Oct 31 20:34:36.204778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.204605    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-xks-ca\") pod \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") "
Oct 31 20:34:36.204778 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.204750    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540ee8ab-a2cd-484c-b1e5-23661ea5b251-kube-api-access-qwmfn\" (UniqueName: \"kubernetes.io/projected/540ee8ab-a2cd-484c-b1e5-23661ea5b251-kube-api-access-qwmfn\") pod \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\" (UID: \"540ee8ab-a2cd-484c-b1e5-23661ea5b251\") "
Oct 31 20:34:36.204952 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.204925    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-openshift-ca" pod "540ee8ab-a2cd-484c-b1e5-23661ea5b251" (UID: "540ee8ab-a2cd-484c-b1e5-23661ea5b251"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:34:36.205113 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.205086    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:34:36.207354 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.207326    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-xks-ca" pod "540ee8ab-a2cd-484c-b1e5-23661ea5b251" (UID: "540ee8ab-a2cd-484c-b1e5-23661ea5b251"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:34:36.207446 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.207362    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540ee8ab-a2cd-484c-b1e5-23661ea5b251-kube-api-access-qwmfn" pod "540ee8ab-a2cd-484c-b1e5-23661ea5b251" (UID: "540ee8ab-a2cd-484c-b1e5-23661ea5b251"). InnerVolumeSpecName "kube-api-access-qwmfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:34:36.306013 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.305973    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwmfn\" (UniqueName: \"kubernetes.io/projected/540ee8ab-a2cd-484c-b1e5-23661ea5b251-kube-api-access-qwmfn\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:34:36.306013 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.306008    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540ee8ab-a2cd-484c-b1e5-23661ea5b251-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:34:36.950164 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.950131    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm" event={"ID":"540ee8ab-a2cd-484c-b1e5-23661ea5b251","Type":"ContainerDied","Data":"79a44a62ad6308f4e5c41e9e868c3f70cbf3412d31f4b303b4eea1eb37fc7aed"}
Oct 31 20:34:36.950354 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.950166    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m2lcm"
Oct 31 20:34:36.950354 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:36.950186    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79a44a62ad6308f4e5c41e9e868c3f70cbf3412d31f4b303b4eea1eb37fc7aed"
Oct 31 20:34:38.077781 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:38.077743    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:34:41.528372 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:41.528325    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:34:41.748478 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:41.748443    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:34:45.025109 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:45.025067    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:34:51.705357 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:51.704779    5006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-t8qk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:34:51.705357 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:51.704873    5006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-t8qk4" podUID="1d98cc29-8ed6-4e88-a844-2a0863c5cfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:34:53.006249 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.006211    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"]
Oct 31 20:34:53.006686 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.006629    5006 cpu_manager.go:414] "containerMap: removing container" podUID="540ee8ab-a2cd-484c-b1e5-23661ea5b251" containerName="revoke-keys"
Oct 31 20:34:53.006686 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.006641    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540ee8ab-a2cd-484c-b1e5-23661ea5b251" containerName="revoke-keys"
Oct 31 20:34:53.006790 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.006777    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="540ee8ab-a2cd-484c-b1e5-23661ea5b251" containerName="revoke-keys"
Oct 31 20:34:53.010206 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.010186    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.062997 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.062960    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"]
Oct 31 20:34:53.086618 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.086577    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-d45pd\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.086812 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.086663    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-d45pd\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.086812 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.086786    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldvzr\" (UniqueName: \"kubernetes.io/projected/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-kube-api-access-ldvzr\") pod \"maas-api-revoke-sub-0165d7d2-d45pd\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.111437 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.111391    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:34:53.111437 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.111419    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:34:53.111437 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.111393    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:34:53.111813 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.111465    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpkrx\""
Oct 31 20:34:53.188142 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.188099    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-d45pd\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.188142 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.188150    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldvzr\" (UniqueName: \"kubernetes.io/projected/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-kube-api-access-ldvzr\") pod \"maas-api-revoke-sub-0165d7d2-d45pd\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.188476 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.188216    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-d45pd\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.188889 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.188867    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-d45pd\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.190576 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.190551    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-d45pd\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.245847 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.245810    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldvzr\" (UniqueName: \"kubernetes.io/projected/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-kube-api-access-ldvzr\") pod \"maas-api-revoke-sub-0165d7d2-d45pd\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.323617 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.323511    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:53.694523 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:34:53.694491    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8ecd34_95c1_4b3c_9063_77d32aa98d22.slice/crio-2fa6efdff6843207118b8f87c413effe5429004a93291263f1233bd4899d050e WatchSource:0}: Error finding container 2fa6efdff6843207118b8f87c413effe5429004a93291263f1233bd4899d050e: Status 404 returned error can't find the container with id 2fa6efdff6843207118b8f87c413effe5429004a93291263f1233bd4899d050e
Oct 31 20:34:53.696096 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:53.696071    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"]
Oct 31 20:34:54.076753 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:54.076711    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd" event={"ID":"ea8ecd34-95c1-4b3c-9063-77d32aa98d22","Type":"ContainerStarted","Data":"67ea3b13431beb3d6c03f3b5aebe1203b9ac9c1ecd3b49d33854a380f954cfda"}
Oct 31 20:34:54.077159 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:54.076769    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd" event={"ID":"ea8ecd34-95c1-4b3c-9063-77d32aa98d22","Type":"ContainerStarted","Data":"2fa6efdff6843207118b8f87c413effe5429004a93291263f1233bd4899d050e"}
Oct 31 20:34:54.192462 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:54.192410    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd" podStartSLOduration=2.192397765 podStartE2EDuration="2.192397765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:34:54.19037443 +0000 UTC m=+4712.719908483" watchObservedRunningTime="2026-10-31 20:34:54.192397765 +0000 UTC m=+4712.721931838"
Oct 31 20:34:55.084028 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:55.083989    5006 generic.go:356] "Generic (PLEG): container finished" podID="ea8ecd34-95c1-4b3c-9063-77d32aa98d22" containerID="67ea3b13431beb3d6c03f3b5aebe1203b9ac9c1ecd3b49d33854a380f954cfda" exitCode=0
Oct 31 20:34:55.084028 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:55.084032    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd" event={"ID":"ea8ecd34-95c1-4b3c-9063-77d32aa98d22","Type":"ContainerDied","Data":"67ea3b13431beb3d6c03f3b5aebe1203b9ac9c1ecd3b49d33854a380f954cfda"}
Oct 31 20:34:56.230850 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:56.230817    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:56.320285 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:56.320246    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-openshift-ca\") pod \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") "
Oct 31 20:34:56.320467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:56.320304    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-kube-api-access-ldvzr\" (UniqueName: \"kubernetes.io/projected/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-kube-api-access-ldvzr\") pod \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") "
Oct 31 20:34:56.320467 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:56.320454    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-xks-ca\") pod \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\" (UID: \"ea8ecd34-95c1-4b3c-9063-77d32aa98d22\") "
Oct 31 20:34:56.320626 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:56.320602    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-openshift-ca" pod "ea8ecd34-95c1-4b3c-9063-77d32aa98d22" (UID: "ea8ecd34-95c1-4b3c-9063-77d32aa98d22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:34:56.320730 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:56.320716    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:34:56.322590 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:56.322564    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-kube-api-access-ldvzr" pod "ea8ecd34-95c1-4b3c-9063-77d32aa98d22" (UID: "ea8ecd34-95c1-4b3c-9063-77d32aa98d22"). InnerVolumeSpecName "kube-api-access-ldvzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:34:56.322641 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:56.322628    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-xks-ca" pod "ea8ecd34-95c1-4b3c-9063-77d32aa98d22" (UID: "ea8ecd34-95c1-4b3c-9063-77d32aa98d22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:34:56.422193 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:56.422104    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:34:56.422193 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:56.422138    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldvzr\" (UniqueName: \"kubernetes.io/projected/ea8ecd34-95c1-4b3c-9063-77d32aa98d22-kube-api-access-ldvzr\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:34:57.096820 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:57.096783    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd" event={"ID":"ea8ecd34-95c1-4b3c-9063-77d32aa98d22","Type":"ContainerDied","Data":"2fa6efdff6843207118b8f87c413effe5429004a93291263f1233bd4899d050e"}
Oct 31 20:34:57.096820 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:57.096821    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d45pd"
Oct 31 20:34:57.097100 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:34:57.096833    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fa6efdff6843207118b8f87c413effe5429004a93291263f1233bd4899d050e"
Oct 31 20:35:04.690687 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.690584    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"]
Oct 31 20:35:04.693138 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.691033    5006 cpu_manager.go:414] "containerMap: removing container" podUID="ea8ecd34-95c1-4b3c-9063-77d32aa98d22" containerName="revoke-keys"
Oct 31 20:35:04.693138 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.691044    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8ecd34-95c1-4b3c-9063-77d32aa98d22" containerName="revoke-keys"
Oct 31 20:35:04.693138 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.691126    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8ecd34-95c1-4b3c-9063-77d32aa98d22" containerName="revoke-keys"
Oct 31 20:35:04.694186 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.694169    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:04.748570 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.748532    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"]
Oct 31 20:35:04.796631 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.796597    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:35:04.796828 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.796707    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:35:04.797363 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.797343    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:35:04.797513 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.797501    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpkrx\""
Oct 31 20:35:04.803895 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.803864    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-q4rh5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:04.804047 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.803942    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-q4rh5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:04.804200 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.804073    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzthl\" (UniqueName: \"kubernetes.io/projected/6af8d093-200a-4345-9c45-c57a9efa50a5-kube-api-access-rzthl\") pod \"maas-api-revoke-sub-6b19338f-q4rh5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:04.904953 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.904914    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-q4rh5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:04.905145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.904991    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-q4rh5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:04.905145 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.905067    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzthl\" (UniqueName: \"kubernetes.io/projected/6af8d093-200a-4345-9c45-c57a9efa50a5-kube-api-access-rzthl\") pod \"maas-api-revoke-sub-6b19338f-q4rh5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:04.905787 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.905753    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-q4rh5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:04.907537 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.907512    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-q4rh5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:04.962731 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:04.962642    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzthl\" (UniqueName: \"kubernetes.io/projected/6af8d093-200a-4345-9c45-c57a9efa50a5-kube-api-access-rzthl\") pod \"maas-api-revoke-sub-6b19338f-q4rh5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:05.006894 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:05.006853    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:05.374482 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:35:05.374435    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af8d093_200a_4345_9c45_c57a9efa50a5.slice/crio-cd7029b12e3ea050c69aab3799e67e484c4938342bafed61091612e35ec1f32f WatchSource:0}: Error finding container cd7029b12e3ea050c69aab3799e67e484c4938342bafed61091612e35ec1f32f: Status 404 returned error can't find the container with id cd7029b12e3ea050c69aab3799e67e484c4938342bafed61091612e35ec1f32f
Oct 31 20:35:05.375928 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:05.375900    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"]
Oct 31 20:35:05.398196 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:35:05.398159    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:35:06.142970 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:06.142931    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5" event={"ID":"6af8d093-200a-4345-9c45-c57a9efa50a5","Type":"ContainerStarted","Data":"7f0d0c1eaa41650ff06f96ce0ba0bec2af67e4138c0da0d68478f288b5e9d7f9"}
Oct 31 20:35:06.143392 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:06.142988    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5" event={"ID":"6af8d093-200a-4345-9c45-c57a9efa50a5","Type":"ContainerStarted","Data":"cd7029b12e3ea050c69aab3799e67e484c4938342bafed61091612e35ec1f32f"}
Oct 31 20:35:06.259014 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:06.258557    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5" podStartSLOduration=2.258541497 podStartE2EDuration="2.258541497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:35:06.257559488 +0000 UTC m=+4724.787093540" watchObservedRunningTime="2026-10-31 20:35:06.258541497 +0000 UTC m=+4724.788075550"
Oct 31 20:35:07.150306 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:07.150265    5006 generic.go:356] "Generic (PLEG): container finished" podID="6af8d093-200a-4345-9c45-c57a9efa50a5" containerID="7f0d0c1eaa41650ff06f96ce0ba0bec2af67e4138c0da0d68478f288b5e9d7f9" exitCode=0
Oct 31 20:35:07.150758 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:07.150358    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5" event={"ID":"6af8d093-200a-4345-9c45-c57a9efa50a5","Type":"ContainerDied","Data":"7f0d0c1eaa41650ff06f96ce0ba0bec2af67e4138c0da0d68478f288b5e9d7f9"}
Oct 31 20:35:08.310370 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:08.310338    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:08.440419 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:08.440328    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-openshift-ca\") pod \"6af8d093-200a-4345-9c45-c57a9efa50a5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") "
Oct 31 20:35:08.440419 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:08.440386    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af8d093-200a-4345-9c45-c57a9efa50a5-kube-api-access-rzthl\" (UniqueName: \"kubernetes.io/projected/6af8d093-200a-4345-9c45-c57a9efa50a5-kube-api-access-rzthl\") pod \"6af8d093-200a-4345-9c45-c57a9efa50a5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") "
Oct 31 20:35:08.440700 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:08.440537    5006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-xks-ca\") pod \"6af8d093-200a-4345-9c45-c57a9efa50a5\" (UID: \"6af8d093-200a-4345-9c45-c57a9efa50a5\") "
Oct 31 20:35:08.440805 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:08.440778    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-openshift-ca" pod "6af8d093-200a-4345-9c45-c57a9efa50a5" (UID: "6af8d093-200a-4345-9c45-c57a9efa50a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:35:08.440893 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:08.440876    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-openshift-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:35:08.442608 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:08.442578    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-xks-ca" pod "6af8d093-200a-4345-9c45-c57a9efa50a5" (UID: "6af8d093-200a-4345-9c45-c57a9efa50a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:35:08.442770 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:08.442675    5006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af8d093-200a-4345-9c45-c57a9efa50a5-kube-api-access-rzthl" pod "6af8d093-200a-4345-9c45-c57a9efa50a5" (UID: "6af8d093-200a-4345-9c45-c57a9efa50a5"). InnerVolumeSpecName "kube-api-access-rzthl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:35:08.541393 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:08.541355    5006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzthl\" (UniqueName: \"kubernetes.io/projected/6af8d093-200a-4345-9c45-c57a9efa50a5-kube-api-access-rzthl\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:35:08.541393 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:08.541387    5006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af8d093-200a-4345-9c45-c57a9efa50a5-maas-api-xks-ca\") on node \"ip-10-0-128-104.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 20:35:09.162355 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:09.162322    5006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5"
Oct 31 20:35:09.162548 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:09.162361    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-q4rh5" event={"ID":"6af8d093-200a-4345-9c45-c57a9efa50a5","Type":"ContainerDied","Data":"cd7029b12e3ea050c69aab3799e67e484c4938342bafed61091612e35ec1f32f"}
Oct 31 20:35:09.162548 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:09.162416    5006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd7029b12e3ea050c69aab3799e67e484c4938342bafed61091612e35ec1f32f"
Oct 31 20:35:16.770998 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:16.770963    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccc6db5f7-5g26l_86cd364e-be8d-48bb-a86d-fb588b2fd6c9/authorino/0.log"
Oct 31 20:35:17.346793 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:17.346754    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6qj5r"]
Oct 31 20:35:18.882119 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:18.882077    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:35:28.193583 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:28.193534    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pfnsg_d53b3837-5749-40ea-956b-344928900061/manager/0.log"
Oct 31 20:35:28.555518 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:28.555446    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-knn9s_c826795e-be35-4ba7-82c9-0d9e634d886d/manager/0.log"
Oct 31 20:35:29.696159 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:29.696128    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5bdnl_51ed068b-24d8-49f9-a605-66d9499b2e4f/manager/0.log"
Oct 31 20:35:31.230081 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:31.230047    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-d45pd_ea8ecd34-95c1-4b3c-9063-77d32aa98d22/revoke-keys/0.log"
Oct 31 20:35:31.442204 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:31.442168    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-q4rh5_6af8d093-200a-4345-9c45-c57a9efa50a5/revoke-keys/0.log"
Oct 31 20:35:31.663661 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:31.663630    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-m2lcm_540ee8ab-a2cd-484c-b1e5-23661ea5b251/revoke-keys/0.log"
Oct 31 20:35:33.349665 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:33.349619    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccc6db5f7-5g26l_86cd364e-be8d-48bb-a86d-fb588b2fd6c9/authorino/0.log"
Oct 31 20:35:33.783126 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:33.783041    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k26bt_e3cbeb55-7b7e-462f-887f-c88b6a6f4b59/manager/0.log"
Oct 31 20:35:34.003779 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:34.003742    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mzlv4_3b1eee4c-644d-41e7-b954-ec9f227b53dc/manager/0.log"
Oct 31 20:35:34.973303 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:34.973270    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6qj5r_aefb9b87-e1f9-47f4-ad88-17f86708bacf/limitador/0.log"
Oct 31 20:35:35.194142 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:35.194110    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-zxk9k_68655beb-12b2-4ba2-a31f-7542299aeda4/manager/0.log"
Oct 31 20:35:35.393570 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:35:35.393528    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:35:36.007187 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:36.007156    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-69f549cf5c44lqr_2685eee1-8b37-4d0b-8003-8e85186cc299/istio-proxy/0.log"
Oct 31 20:35:37.150694 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:37.150642    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-cccdj_b14112fb-b3f3-4ba3-8e12-af608d6c6beb/istio-proxy/0.log"
Oct 31 20:35:38.405404 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:38.405366    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:35:38.547913 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:38.547882    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-t8qk4_1d98cc29-8ed6-4e88-a844-2a0863c5cfdf/istio-proxy/0.log"
Oct 31 20:35:38.797767 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:38.797691    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-vqn2k_421392b6-59fb-4d6d-a35f-acfd09346096/istio-proxy/0.log"
Oct 31 20:35:39.224698 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:39.224668    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr_b0383d77-c63f-4507-aa64-b080e2a16a00/istio-proxy/0.log"
Oct 31 20:35:39.872708 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:39.872669    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddc6dfd67-fk8lv_f9c1e13d-3f76-4884-af1a-956dc34fd1d6/router/0.log"
Oct 31 20:35:43.290285 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:43.290254    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccc6db5f7-5g26l_86cd364e-be8d-48bb-a86d-fb588b2fd6c9/authorino/0.log"
Oct 31 20:35:52.833578 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:52.833539    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pfnsg_d53b3837-5749-40ea-956b-344928900061/manager/0.log"
Oct 31 20:35:53.190835 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:53.190799    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-knn9s_c826795e-be35-4ba7-82c9-0d9e634d886d/manager/0.log"
Oct 31 20:35:54.056870 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:54.056840    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5bdnl_51ed068b-24d8-49f9-a605-66d9499b2e4f/manager/0.log"
Oct 31 20:35:55.411997 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:55.411969    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-d45pd_ea8ecd34-95c1-4b3c-9063-77d32aa98d22/revoke-keys/0.log"
Oct 31 20:35:55.627541 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:55.627507    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-q4rh5_6af8d093-200a-4345-9c45-c57a9efa50a5/revoke-keys/0.log"
Oct 31 20:35:55.844565 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:55.844540    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-m2lcm_540ee8ab-a2cd-484c-b1e5-23661ea5b251/revoke-keys/0.log"
Oct 31 20:35:57.160425 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:57.160384    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccc6db5f7-5g26l_86cd364e-be8d-48bb-a86d-fb588b2fd6c9/authorino/0.log"
Oct 31 20:35:57.599717 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:57.599681    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k26bt_e3cbeb55-7b7e-462f-887f-c88b6a6f4b59/manager/0.log"
Oct 31 20:35:57.818294 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:57.818257    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mzlv4_3b1eee4c-644d-41e7-b954-ec9f227b53dc/manager/0.log"
Oct 31 20:35:58.674924 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:58.674891    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6qj5r_aefb9b87-e1f9-47f4-ad88-17f86708bacf/limitador/0.log"
Oct 31 20:35:58.883710 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:58.883675    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-zxk9k_68655beb-12b2-4ba2-a31f-7542299aeda4/manager/0.log"
Oct 31 20:35:59.501837 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:35:59.501806    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-69f549cf5c44lqr_2685eee1-8b37-4d0b-8003-8e85186cc299/istio-proxy/0.log"
Oct 31 20:36:00.662784 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:36:00.662749    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-cccdj_b14112fb-b3f3-4ba3-8e12-af608d6c6beb/istio-proxy/0.log"
Oct 31 20:36:02.041589 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:36:02.041548    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-t8qk4_1d98cc29-8ed6-4e88-a844-2a0863c5cfdf/istio-proxy/0.log"
Oct 31 20:36:02.313173 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:36:02.313079    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-vqn2k_421392b6-59fb-4d6d-a35f-acfd09346096/istio-proxy/0.log"
Oct 31 20:36:02.793665 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:36:02.793626    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-649845f967-p7xbr_b0383d77-c63f-4507-aa64-b080e2a16a00/istio-proxy/0.log"
Oct 31 20:36:03.463800 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:36:03.463737    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddc6dfd67-fk8lv_f9c1e13d-3f76-4884-af1a-956dc34fd1d6/router/0.log"
Oct 31 20:36:05.393280 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:36:05.393242    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:36:22.410115 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:36:22.410084    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:36:22.488151 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:36:22.488117    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nztjc_a3897285-e4e6-45da-bf32-fcadd1f9b506/console-operator/2.log"
Oct 31 20:36:35.394355 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:36:35.394254    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:36:38.743966 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:36:38.743929    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:37:05.396263 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:37:05.393841    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:37:12.394463 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:12.394420    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:37:14.520326 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:14.520288    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:37:24.634000 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:24.633965    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vs8zn_0c39943b-0a43-4184-b616-dda8d9f8545e/global-pull-secret-syncer/0.log"
Oct 31 20:37:25.227191 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:25.227160    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mjspt_78ed8e27-2cd2-43e5-8ee1-23042619743e/connection-checker/0.log"
Oct 31 20:37:25.591741 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:25.591714    5006 scope.go:122] "RemoveContainer" containerID="e2de97fa97457b8b036438baabd1df044be3a5c47114e11a9d04a80c2f6e94f9"
Oct 31 20:37:25.633325 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:25.633298    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vmhw6_291cf47e-f759-4489-abb5-1bafffff7b25/connection-checker/0.log"
Oct 31 20:37:26.039959 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:26.039880    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wq4hz_73377bc4-2624-4ed6-a5aa-30228b89cbfb/connection-checker/0.log"
Oct 31 20:37:26.479115 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:26.479077    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log"
Oct 31 20:37:27.686568 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:27.686541    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-104.us-west-2.compute.internal_70cf5f03a5aad2f217175f2d65705fc5/haproxy/0.log"
Oct 31 20:37:31.938405 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:31.938359    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccc6db5f7-5g26l_86cd364e-be8d-48bb-a86d-fb588b2fd6c9/authorino/0.log"
Oct 31 20:37:32.247186 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.247094    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k26bt_e3cbeb55-7b7e-462f-887f-c88b6a6f4b59/manager/0.log"
Oct 31 20:37:32.633138 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.633108    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mzlv4_3b1eee4c-644d-41e7-b954-ec9f227b53dc/manager/0.log"
Oct 31 20:37:32.697597 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.697564    5006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"]
Oct 31 20:37:32.698050 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.698036    5006 cpu_manager.go:414] "containerMap: removing container" podUID="6af8d093-200a-4345-9c45-c57a9efa50a5" containerName="revoke-keys"
Oct 31 20:37:32.698095 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.698051    5006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af8d093-200a-4345-9c45-c57a9efa50a5" containerName="revoke-keys"
Oct 31 20:37:32.698146 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.698137    5006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af8d093-200a-4345-9c45-c57a9efa50a5" containerName="revoke-keys"
Oct 31 20:37:32.701256 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.701240    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.755827 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.755797    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"]
Oct 31 20:37:32.760851 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.760823    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-podres\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.761001 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.760866    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-lib-modules\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.761076 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.761047    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59vhw\" (UniqueName: \"kubernetes.io/projected/8e6018bb-880a-4892-8bbb-38c344a22800-kube-api-access-59vhw\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.761203 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.761183    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-proc\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.761255 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.761236    5006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-sys\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.803464 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.803422    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ks7ld\"/\"kube-root-ca.crt\""
Oct 31 20:37:32.803668 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.803472    5006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ks7ld\"/\"openshift-service-ca.crt\""
Oct 31 20:37:32.804954 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.804940    5006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ks7ld\"/\"default-dockercfg-lbbt2\""
Oct 31 20:37:32.862680 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.862625    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-proc\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.862866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.862693    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-sys\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.862866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.862736    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-podres\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.862866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.862769    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-lib-modules\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.862866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.862775    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-proc\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.862866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.862781    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-sys\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.862866 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.862835    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-podres\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.863085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.862869    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e6018bb-880a-4892-8bbb-38c344a22800-lib-modules\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.863085 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.862894    5006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59vhw\" (UniqueName: \"kubernetes.io/projected/8e6018bb-880a-4892-8bbb-38c344a22800-kube-api-access-59vhw\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:32.920863 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:32.920787    5006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59vhw\" (UniqueName: \"kubernetes.io/projected/8e6018bb-880a-4892-8bbb-38c344a22800-kube-api-access-59vhw\") pod \"perf-node-gather-daemonset-n9hwc\" (UID: \"8e6018bb-880a-4892-8bbb-38c344a22800\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:33.014473 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:33.014440    5006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:33.380723 ip-10-0-128-104 kubenswrapper[5006]: W1031 20:37:33.380691    5006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8e6018bb_880a_4892_8bbb_38c344a22800.slice/crio-411512a6bc0ba3e3152039f27d0f29ce8ce2e2216485c30574777fea742b60ac WatchSource:0}: Error finding container 411512a6bc0ba3e3152039f27d0f29ce8ce2e2216485c30574777fea742b60ac: Status 404 returned error can't find the container with id 411512a6bc0ba3e3152039f27d0f29ce8ce2e2216485c30574777fea742b60ac
Oct 31 20:37:33.383254 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:33.383226    5006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"]
Oct 31 20:37:33.839845 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:33.839811    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6qj5r_aefb9b87-e1f9-47f4-ad88-17f86708bacf/limitador/0.log"
Oct 31 20:37:33.929864 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:33.929820    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc" event={"ID":"8e6018bb-880a-4892-8bbb-38c344a22800","Type":"ContainerStarted","Data":"4461eadda53f72b673829ece664a4d9de6eeb64eb26c3d4d1b76c77b20ef4ef5"}
Oct 31 20:37:33.930072 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:33.929882    5006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc" event={"ID":"8e6018bb-880a-4892-8bbb-38c344a22800","Type":"ContainerStarted","Data":"411512a6bc0ba3e3152039f27d0f29ce8ce2e2216485c30574777fea742b60ac"}
Oct 31 20:37:33.930072 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:33.929909    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:34.044821 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:34.044764    5006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc" podStartSLOduration=2.044745601 podStartE2EDuration="2.044745601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 20:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:37:34.043054502 +0000 UTC m=+4872.572588554" watchObservedRunningTime="2026-10-31 20:37:34.044745601 +0000 UTC m=+4872.574279654"
Oct 31 20:37:34.276184 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:34.276074    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-zxk9k_68655beb-12b2-4ba2-a31f-7542299aeda4/manager/0.log"
Oct 31 20:37:35.393138 ip-10-0-128-104 kubenswrapper[5006]: E1031 20:37:35.393089    5006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnscp" containerName="extractor"
Oct 31 20:37:39.141938 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:39.141902    5006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kttjn_b05731a8-35c3-4afa-97fa-6d3db50533b3/konnectivity-agent/0.log" limit=1024
Oct 31 20:37:39.410456 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:39.410367    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-9pglj_b1e328bc-df61-43b5-a5cf-12a2389903a6/cluster-monitoring-operator/0.log"
Oct 31 20:37:39.946384 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:39.946355    5006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-n9hwc"
Oct 31 20:37:44.121772 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:44.121744    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nllvs_20fab3dd-c950-4c2e-b47a-f1dd3c9b1105/node-exporter/0.log"
Oct 31 20:37:44.505520 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:44.505443    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nllvs_20fab3dd-c950-4c2e-b47a-f1dd3c9b1105/kube-rbac-proxy/0.log"
Oct 31 20:37:44.889282 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:44.889251    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nllvs_20fab3dd-c950-4c2e-b47a-f1dd3c9b1105/init-textfile/0.log"
Oct 31 20:37:49.965080 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:49.965050    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-5qfg4_62ce7d89-451a-4b49-800d-b803eba6758c/prometheus-operator-admission-webhook/0.log"
Oct 31 20:37:54.295626 ip-10-0-128-104 kubenswrapper[5006]: I1031 20:37:54.295589    5006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-69f549cf5c44lqr_2685eee1-8b37-4d0b-8003-8e85186cc299/istio-proxy/0.log"